Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes

3 May 2023

Lokesh Nagalapatti

Chun-Liang Li

Chih-Kuan Yeh

Hootan Nakhost

Yasuhisa Fujii

Alexander Ratner

Ranjay Krishna

Chen-Yu Lee

Tomas Pfister

ALM

ArXiv PDF HTML

Papers citing "Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes"

26 / 76 papers shown

Title
Can Small Language Models be Good Reasoners for Sequential Recommendation? Yuling Wang Changxin Tian Binbin Hu Yanhua Yu Ziqi Liu Zhiqiang Zhang Jun Zhou Liang Pang Xiao Wang LRM 40 25 0 07 Mar 2024
Learning to Maximize Mutual Information for Chain-of-Thought Distillation Xin Chen Hanxian Huang Yanjun Gao Yi Wang Jishen Zhao Ke Ding 35 11 0 05 Mar 2024
GenAINet: Enabling Wireless Collective Intelligence via Knowledge Transfer and Reasoning Han Zou Qiyang Zhao Lina Bariah Yu Tian M. Bennis S. Lasaulce 91 12 0 26 Feb 2024
Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes Lucio Dery Steven Kolawole Jean-Francois Kagey Virginia Smith Graham Neubig Ameet Talwalkar 39 28 0 08 Feb 2024
Knowledge Fusion of Large Language Models Fanqi Wan Xinting Huang Deng Cai Xiaojun Quan Wei Bi Shuming Shi MoMe 29 61 0 19 Jan 2024
Know Your Needs Better: Towards Structured Understanding of Marketer Demands with Analogical Reasoning Augmented LLMs Junjie Wang Dan Yang Binbin Hu Yue Shen Wen Zhang Jinjie Gu LRM 35 0 0 09 Jan 2024
ALMANACS: A Simulatability Benchmark for Language Model Explainability Edmund Mills Shiye Su Stuart J. Russell Scott Emmons 46 7 0 20 Dec 2023
Building a Llama2-finetuned LLM for Odia Language Utilizing Domain Knowledge Instruction Set Guneet Singh Kohli Shantipriya Parida Sambit Sekhar Samirit Saha Nipun B. Nair Parul Agarwal Sonal Khosla Kusumlata Patiyal Debasish Dhal 30 13 0 19 Dec 2023
Mixed Distillation Helps Smaller Language Model Better Reasoning Chenglin Li Qianglong Chen Liangyue Li Wang Caiyu Yicheng Li Zhang Yin Yin Zhang LRM 30 11 0 17 Dec 2023
Efficient Toxic Content Detection by Bootstrapping and Distilling Large Language Models Jiang Zhang Qiong Wu Yiming Xu Cheng Cao Zheng Du Konstantinos Psounis 28 14 0 13 Dec 2023
From Classification to Clinical Insights: Towards Analyzing and Reasoning About Mobile and Behavioral Health Data With Large Language Models Zachary Englhardt Chengqian Ma Margaret E. Morris X. Xu Chun-Cheng Chang Lianhui Qin Daniel J. McDuff Xin Liu Shwetak N. Patel Vikram Iyer AI4MH 39 11 0 21 Nov 2023
Explain-then-Translate: An Analysis on Improving Program Translation with Self-generated Explanations Zilu Tang Mayank Agarwal Alex Shypula Bailin Wang Derry Wijaya Jie Chen Yoon Kim LRM 35 15 0 13 Nov 2023
Let's Synthesize Step by Step: Iterative Dataset Synthesis with Large Language Models by Extrapolating Errors from Small Models Ruida Wang Wangchunshu Zhou Mrinmaya Sachan 19 32 0 20 Oct 2023
Sci-CoT: Leveraging Large Language Models for Enhanced Knowledge Distillation in Small Models for Scientific QA Yuhan Ma Haiqi Jiang Chenyou Fan LRM 26 14 0 09 Aug 2023
Learning Evaluation Models from Large Language Models for Sequence Generation Chenglong Wang Hang Zhou Kai-Chun Chang Tongran Liu Chunliang Zhang Quan Du Tong Xiao Yue Zhang Jingbo Zhu ELM 34 3 0 08 Aug 2023
UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition Wenxuan Zhou Sheng Zhang Yu Gu Muhao Chen Hoifung Poon 22 59 0 07 Aug 2023
Passive learning of active causal strategies in agents and language models Andrew Kyle Lampinen Stephanie C. Y. Chan Ishita Dasgupta A. Nam Jane X. Wang 29 15 0 25 May 2023
The False Promise of Imitating Proprietary LLMs Arnav Gudibande Eric Wallace Charles Burton Snell Xinyang Geng Hao Liu Pieter Abbeel Sergey Levine Dawn Song ALM 27 196 0 25 May 2023
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model Jacob Eisenstein D. Andor Bernd Bohnet Michael Collins David M. Mimno LRM 189 24 0 05 Oct 2022
Ask Me Anything: A simple strategy for prompting language models Simran Arora A. Narayan Mayee F. Chen Laurel J. Orr Neel Guha Kush S. Bhatia Ines Chami Frederic Sala Christopher Ré ReLM LRM 211 206 0 05 Oct 2022
Large Language Models are Zero-Shot Reasoners Takeshi Kojima S. Gu Machel Reid Yutaka Matsuo Yusuke Iwasawa ReLM LRM 307 4,077 0 24 May 2022
Self-Consistency Improves Chain of Thought Reasoning in Language Models Xuezhi Wang Jason W. Wei Dale Schuurmans Quoc Le Ed H. Chi Sharan Narang Aakanksha Chowdhery Denny Zhou ReLM BDL LRM AI4CE 314 3,237 0 21 Mar 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 347 8,457 0 28 Jan 2022
The Power of Scale for Parameter-Efficient Prompt Tuning Brian Lester Rami Al-Rfou Noah Constant VPVLM 280 3,844 0 18 Apr 2021
Measuring Association Between Labels and Free-Text Rationales Sarah Wiegreffe Ana Marasović Noah A. Smith 274 170 0 24 Oct 2020
e-SNLI: Natural Language Inference with Natural Language Explanations Oana-Maria Camburu Tim Rocktaschel Thomas Lukasiewicz Phil Blunsom LRM 255 620 0 04 Dec 2018