Compact Language Models via Pruning and Knowledge Distillation

19 July 2024

Saurav Muralidharan

Sharath Turuvekere Sreenivas

Papers citing "Compact Language Models via Pruning and Knowledge Distillation"

33 / 33 papers shown

Title
Flash-VL 2B: Optimizing Vision-Language Model Performance for Ultra-Low Latency and High Throughput Bo Zhang Shuo Li Runhe Tian Yang Yang Jixin Tang Jinhao Zhou Lin Ma VLM 27 0 0 14 May 2025
ReplaceMe: Network Simplification via Layer Pruning and Linear Transformations Dmitriy Shopkhoev Ammar Ali Magauiya Zhussip Valentin Malykh Stamatios Lefkimmiatis N. Komodakis Sergey Zagoruyko VLM 131 0 0 05 May 2025
ConTextual: Improving Clinical Text Summarization in LLMs with Context-preserving Token Filtering and Knowledge Graphs Fahmida Liza Piya Rahmatollah Beheshti 131 0 0 23 Apr 2025
Cat, Rat, Meow: On the Alignment of Language Model and Human Term-Similarity Judgments Lorenz Linhardt Tom Neuhäuser Lenka Tětková Oliver Eberle ALM AI4TS 42 0 0 10 Apr 2025
Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models Ruikang Liu Yuxuan Sun Manyi Zhang Haoli Bai Xianzhi Yu Tiezheng Yu C. Yuan Lu Hou MQ LRM 31 5 0 07 Apr 2025
Overcoming Vocabulary Mismatch: Vocabulary-agnostic Teacher Guided Language Modeling Haebin Shin Lei Ji Xiao Liu Yeyun Gong 52 0 0 24 Mar 2025
PLM: Efficient Peripheral Language Models Hardware-Co-Designed for Ubiquitous Computing Cheng Deng Luoyang Sun Jiwen Jiang Yongcheng Zeng Xinjian Wu ... Haoyang Li Lei Chen Lionel M. Ni H. Zhang Jun Wang 160 0 0 15 Mar 2025
IDEA Prune: An Integrated Enlarge-and-Prune Pipeline in Generative Language Model Pretraining Yixiao Li Xianzhi Du Ajay Jaiswal Tao Lei T. Zhao Chong-Jun Wang Jianyu Wang 50 1 0 07 Mar 2025
Kanana: Compute-efficient Bilingual Language Models Kanana LLM Team Yunju Bak Hojin Lee Minho Ryu Jiyeon Ham ... Daniel Lee Minchul Lee M. Lee Shinbok Lee Gaeun Seo 90 1 0 26 Feb 2025
The Curse of Depth in Large Language Models Wenfang Sun Xinyuan Song Pengxiang Li Lu Yin Yefeng Zheng Shiwei Liu 67 4 0 09 Feb 2025
TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models Makoto Shing Kou Misaki Han Bao Sho Yokoi Takuya Akiba VLM 57 1 0 28 Jan 2025
CURing Large Models: Compression via CUR Decomposition Sanghyeon Park Soo-Mook Moon 41 0 0 08 Jan 2025
VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models Byung-Kwan Lee Ryo Hachiuma Yu-Chiang Frank Wang Y. Ro Yueh-Hua Wu VLM 81 0 0 02 Dec 2024
Puzzle: Distillation-Based NAS for Inference-Optimized LLMs Akhiad Bercovich Tomer Ronen Talor Abramovich Nir Ailon Nave Assaf ... Ido Shahaf Oren Tropp Omer Ullman Argov Ran Zilberstein Ran El-Yaniv 77 1 0 28 Nov 2024
Reassessing Layer Pruning in LLMs: New Insights and Methods Yao Lu Hao Cheng Yujie Fang Zeyu Wang Jiaheng Wei Dongwei Xu Qi Xuan Xiaoniu Yang Zhaowei Zhu 63 0 0 23 Nov 2024
GPT for Games: An Updated Scoping Review (2020-2024) Daijin Yang Erica Kleinman Casper Harteveld LLMAG AI4TS AI4CE 48 3 0 01 Nov 2024
BitStack: Any-Size Compression of Large Language Models in Variable Memory Environments Xinghao Wang Pengyu Wang Bo Wang Dong Zhang Yunhua Zhou Xipeng Qiu MQ 36 2 0 31 Oct 2024
Computational Bottlenecks of Training Small-scale Large Language Models Saleh Ashkboos Iman Mirzadeh Keivan Alizadeh Mohammad Hossein Sekhavat Moin Nabi Mehrdad Farajtabar Fartash Faghri 21 0 0 25 Oct 2024
MiniPLM: Knowledge Distillation for Pre-Training Language Models Yuxian Gu Hao Zhou Fandong Meng Jie Zhou Minlie Huang 67 5 0 22 Oct 2024
MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router Yanyue Xie Zhi Zhang Ding Zhou Cong Xie Ziang Song Xin Liu Yanzhi Wang Xue Lin An Xu LLMAG 38 3 0 15 Oct 2024
BlackDAN: A Black-Box Multi-Objective Approach for Effective and Contextual Jailbreaking of Large Language Models Xinyuan Wang Victor Shea-Jay Huang Renmiao Chen Hao Wang C. Pan Lei Sha Minlie Huang AAML 25 2 0 13 Oct 2024
Compressing Large Language Models with Automated Sub-Network Search R. Sukthanker B. Staffler Frank Hutter Aaron Klein LRM 38 0 0 09 Oct 2024
Leveraging Large Language Models for Suicide Detection on Social Media with Limited Labels Vy Nguyen Chau Pham ALM AI4MH 29 2 0 06 Oct 2024
Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling Hritik Bansal Arian Hosseini Rishabh Agarwal Vinh Q. Tran Mehran Kazemi SyDa OffRL LRM 37 37 0 29 Aug 2024
Cross-Domain Foundation Model Adaptation: Pioneering Computer Vision Models for Geophysical Data Analysis Zhixiang Guo Xinming Wu Luming Liang Hanlin Sheng Nuo Chen Zhengfa Bi AI4CE 49 1 0 22 Aug 2024
LLM Pruning and Distillation in Practice: The Minitron Approach Sharath Turuvekere Sreenivas Saurav Muralidharan Raviraj Joshi Marcin Chochowski M. Patwary M. Shoeybi Bryan Catanzaro Jan Kautz Pavlo Molchanov 28 25 0 21 Aug 2024
Model Compression and Efficient Inference for Large Language Models: A Survey Wenxiao Wang Wei Chen Yicong Luo Yongliu Long Zhengkai Lin Liye Zhang Binbin Lin Deng Cai Xiaofei He MQ 41 47 0 15 Feb 2024
Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes Lucio Dery Steven Kolawole Jean-Francois Kagey Virginia Smith Graham Neubig Ameet Talwalkar 39 28 0 08 Feb 2024
Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methods Bo-Kyeong Kim Geonmin Kim Tae-Ho Kim Thibault Castells Shinkook Choi Junho Shin Hyoung-Kyu Song 62 30 0 05 Feb 2024
SliceGPT: Compress Large Language Models by Deleting Rows and Columns Saleh Ashkboos Maximilian L. Croci Marcelo Gennari do Nascimento Torsten Hoefler James Hensman VLM 129 145 0 26 Jan 2024
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 358 8,495 0 28 Jan 2022
Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks Torsten Hoefler Dan Alistarh Tal Ben-Nun Nikoli Dryden Alexandra Peste MQ 141 684 0 31 Jan 2021
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 245 1,821 0 17 Sep 2019