Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes

3 May 2023

Lokesh Nagalapatti

Chun-Liang Li

Chih-Kuan Yeh

Hootan Nakhost

Yasuhisa Fujii

Alexander Ratner

Ranjay Krishna

Chen-Yu Lee

Tomas Pfister

ALM

ArXiv PDF HTML

Papers citing "Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes"

50 / 76 papers shown

Title
Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniques Sanjay Surendranath Girija Shashank Kapoor Lakshit Arora Dipen Pradhan Aman Raj Ankit Shetgaonkar 52 0 0 05 May 2025
LLM-based Semantic Augmentation for Harmful Content Detection Elyas Meguellati Assaad Zeghina S. Sadiq Gianluca Demartini 34 0 0 22 Apr 2025
Exploiting Edited Large Language Models as General Scientific Optimizers Qitan Lv T. Liu H. Wang 36 0 0 08 Mar 2025
Pub-Guard-LLM: Detecting Fraudulent Biomedical Articles with Reliable Explanations Lihu Chen Shuojie Fu Gabriel Freedman Cemre Zor Guy Martin James Kinross Uddhav Vaghela Ovidiu Serban Francesca Toni DeLMO 63 0 0 21 Feb 2025
Dynamic Low-Rank Sparse Adaptation for Large Language Models Weizhong Huang Yuxin Zhang Xiawu Zheng Y. Liu Jing Lin Yiwu Yao Rongrong Ji 85 1 0 21 Feb 2025
Ten Challenging Problems in Federated Foundation Models Tao Fan Hanlin Gu Xuemei Cao Chee Seng Chan Qian Chen ... Y. Zhang Xiaojin Zhang Zhenzhe Zheng Lixin Fan Qiang Yang FedML 81 4 0 14 Feb 2025
Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation H. Seo Wongi Jeong Jae-sun Seo Se Young Chun 55 0 0 12 Feb 2025
Few-shot LLM Synthetic Data with Distribution Matching Jiyuan Ren Zhaocheng Du Zhihao Wen Qinglin Jia Sunhao Dai Chuhan Wu Zhenhua Dong SyDa 77 0 0 09 Feb 2025
Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs Nicolas Boizard Kevin El Haddad C´eline Hudelot Pierre Colombo 68 14 0 28 Jan 2025
Exploring Iterative Enhancement for Improving Learnersourced Multiple-Choice Question Explanations with Large Language Models Qiming Bao Juho Leinonen A. Peng Wanjun Zhong Gael Gendron Tim Pistotti Alice Huang Paul Denny Michael Witbrock J. Liu AI4Ed LRM 173 1 0 20 Jan 2025
CodEv: An Automated Grading Framework Leveraging Large Language Models for Consistent and Constructive Feedback En-Qi Tseng Pei-Cing Huang Chan Hsu Peng-Yi Wu Chan-Tung Ku Yihuang Kang 41 1 0 10 Jan 2025
The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better Scott Geng Cheng-Yu Hsieh Vivek Ramanujan Matthew Wallingford Chun-Liang Li Pang Wei Koh Ranjay Krishna DiffM 68 6 0 03 Jan 2025
Towards Efficient and Explainable Hate Speech Detection via Model Distillation Paloma Piot Javier Parapar 78 173 0 18 Dec 2024
ClarityEthic: Explainable Moral Judgment Utilizing Contrastive Ethical Insights from Large Language Models Yuxi Sun Wei Gao Jing Ma Hongzhan Lin Ziyang Luo Wenxuan Zhang ELM 82 0 0 17 Dec 2024
On Memorization of Large Language Models in Logical Reasoning Chulin Xie Yangsibo Huang Chiyuan Zhang Da Yu Xinyun Chen Bill Yuchen Lin Bo Li Badih Ghazi Ravi Kumar LRM 51 20 0 30 Oct 2024
Guiding Through Complexity: What Makes Good Supervision for Hard Math Reasoning Tasks? Xuan He Da Yin Nanyun Peng LRM 38 0 0 27 Oct 2024
SWITCH: Studying with Teacher for Knowledge Distillation of Large Language Models Jahyun Koo Yerin Hwang Yongil Kim Taegwan Kang Hyunkyung Bae Kyomin Jung 55 0 0 25 Oct 2024
MiniPLM: Knowledge Distillation for Pre-Training Language Models Yuxian Gu Hao Zhou Fandong Meng Jie Zhou Minlie Huang 65 5 0 22 Oct 2024
ToW: Thoughts of Words Improve Reasoning in Large Language Models Zhikun Xu Ming shen Jacob Dineen Zhaonan Li Xiao Ye Shijie Lu Aswin Rrv Chitta Baral Ben Zhou LRM 121 1 0 21 Oct 2024
Coarse-to-Fine Highlighting: Reducing Knowledge Hallucination in Large Language Models Qitan Lv Jie Wang Hanzhu Chen Bin Li Yongdong Zhang Feng Wu HILM 19 3 0 19 Oct 2024
Boosting LLM Translation Skills without General Ability Loss via Rationale Distillation Junhong Wu Yang Zhao Yangyifan Xu Bing Liu Chengqing Zong CLL 33 1 0 17 Oct 2024
Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling W. Xu Rujun Han Z. Wang L. Le Dhruv Madeka Lei Li W. Wang Rishabh Agarwal Chen-Yu Lee Tomas Pfister 78 8 0 15 Oct 2024
SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration Heming Xia Yongqi Li Jun Zhang Cunxiao Du Wenjie Li LRM 46 5 0 09 Oct 2024
Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models Hui-Po Wang Mario Fritz 33 3 0 26 Sep 2024
Neural-Symbolic Collaborative Distillation: Advancing Small Language Models for Complex Reasoning Tasks Huanxuan Liao Shizhu He Yao Xu Yuanzhe Zhang Kang Liu Jun Zhao LRM 53 3 0 20 Sep 2024
Efficient Knowledge Distillation: Empowering Small Language Models with Teacher Model Insights Mohamad Ballout U. Krumnack Gunther Heidemann Kai-Uwe Kühnberger 31 2 0 19 Sep 2024
What is the Role of Small Models in the LLM Era: A Survey Lihu Chen Gaël Varoquaux ALM 58 23 0 10 Sep 2024
LUK: Empowering Log Understanding with Expert Knowledge from Large Language Models Lipeng Ma Weidong Yang Sihang Jiang Ben Fei Mingjie Zhou Shuhao Li Bo Xu Bo Xu Yanghua Xiao 51 0 0 03 Sep 2024
Real-Time Video Generation with Pyramid Attention Broadcast Xuanlei Zhao Xiaolong Jin Kai Wang Yang You VGen DiffM 74 31 0 22 Aug 2024
Rapid Biomedical Research Classification: The Pandemic PACT Advanced Categorisation Engine Omid Rohanian Mohammadmahdi Nouriborji Olena Seminog Rodrigo Furst Thomas Mendy ... Georgina Humphreys E. Antonio Adrian Bucher Alice Norton David A. Clifton 52 0 0 14 Jul 2024
Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting Zilong Wang Zifeng Wang Long Le Huaixiu Steven Zheng Swaroop Mishra ... Anush Mattapalli Ankur Taly Jingbo Shang Chen-Yu Lee Tomas Pfister RALM 75 31 0 11 Jul 2024
Composable Interventions for Language Models Arinbjorn Kolbeinsson Kyle O'Brien Tianjin Huang Shanghua Gao Shiwei Liu ... Anurag J. Vaidya Faisal Mahmood Marinka Zitnik Tianlong Chen Thomas Hartvigsen KELM MU 82 5 0 09 Jul 2024
Retrieved In-Context Principles from Previous Mistakes Hao-Lun Sun Yong-jia Jiang Bo Wang Yingyan Hou Yan Zhang Pengjun Xie Fei Huang 52 1 0 08 Jul 2024
CAVE: Controllable Authorship Verification Explanations Sahana Ramnath Kartik Pandey Elizabeth Boschee Xiang Ren 59 1 0 24 Jun 2024
$$\texttt{MoE-RBench}$: Towards Building Reliable Language Models with Sparse Mixture-of-Experts$ $\texttt{MoE-RBench}$ : Towards Building Reliable Language Models with Sparse Mixture-of-Experts Guanjie Chen Xinyu Zhao Tianlong Chen Yu Cheng MoE 66 5 0 17 Jun 2024
Self-Training for Sample-Efficient Active Learning for Text Classification with Pre-Trained Language Models Christopher Schröder Gerhard Heyer VLM 42 0 0 13 Jun 2024
DistilDoc: Knowledge Distillation for Visually-Rich Document Applications Jordy Van Landeghem Subhajit Maity Ayan Banerjee Matthew Blaschko Marie-Francine Moens Josep Lladós Sanket Biswas 41 2 0 12 Jun 2024
Improve Student's Reasoning Generalizability through Cascading Decomposed CoTs Distillation Chengwei Dai Kun Li Wei Zhou Song Hu LRM 44 3 0 30 May 2024
Enhancing Reinforcement Learning with Label-Sensitive Reward for Natural Language Understanding Kuo Liao Shuang Li Meng Zhao Liqun Liu Mengge Xue Zhenyu Hu Honglin Han Chengguo Yin 38 1 0 30 May 2024
Beyond Imitation: Learning Key Reasoning Steps from Dual Chain-of-Thoughts in Reasoning Distillation Chengwei Dai Kun Li Wei Zhou Song Hu LRM 36 5 0 30 May 2024
Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Models Byung-Kwan Lee Chae Won Kim Beomchan Park Yonghyun Ro MLLM LRM 27 17 0 24 May 2024
Large Language Models for Cyber Security: A Systematic Literature Review HanXiang Xu Shenao Wang Ningke Li K. Wang Yanjie Zhao Kai Chen Ting Yu Yang Janet Liu H. Wang 31 23 0 08 May 2024
Beyond Chain-of-Thought: A Survey of Chain-of-X Paradigms for LLMs Yu Xia Rui Wang Xu Liu Mingyan Li Tong Yu Xiang Chen Julian McAuley Shuai Li LRM 43 18 0 24 Apr 2024
Describe-then-Reason: Improving Multimodal Mathematical Reasoning through Visual Comprehension Training Mengzhao Jia Zhihan Zhang W. Yu Fangkai Jiao Meng-Long Jiang VLM ReLM LRM 48 7 0 22 Apr 2024
Distilling Reasoning Ability from Large Language Models with Adaptive Thinking Xiao Chen Sihang Zhou K. Liang Xinwang Liu ReLM LRM 29 2 0 14 Apr 2024
Dialogue with Robots: Proposals for Broadening Participation and Research in the SLIVAR Community Casey Kennington Malihe Alikhani Heather Pon-Barry Katherine Atwell Yonatan Bisk ... Jivko Sinapov Angela Stewart Matthew Stone Stefanie Tellex Tom Williams 49 0 0 01 Apr 2024
Constructing Multilingual Visual-Text Datasets Revealing Visual Multilingual Ability of Vision Language Models Jesse Atuhurra Iqra Ali Tatsuya Hiraoka Hidetaka Kamigaito Tomoya Iwakura Taro Watanabe 38 1 0 29 Mar 2024
The Unreasonable Ineffectiveness of the Deeper Layers Andrey Gromov Kushal Tirumala Hassan Shapourian Paolo Glorioso Daniel A. Roberts 41 79 0 26 Mar 2024
SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression Xin Wang Yu Zheng Zhongwei Wan Mi Zhang MQ 55 43 0 12 Mar 2024
The pitfalls of next-token prediction Gregor Bachmann Vaishnavh Nagarajan 35 59 0 11 Mar 2024