Title
Reassessing Layer Pruning in LLMs: New Insights and Methods Yao Lu Hao Cheng Yujie Fang Zeyu Wang Jiaheng Wei Dongwei Xu Qi Xuan Xiaoniu Yang Zhaowei Zhu 130 4 0 23 Nov 2024
MPruner: Optimizing Neural Network Size with CKA-Based Mutual Information Pruning Seungbeom Hu ChanJun Park Andrew Ferraiuolo Sang-Ki Ko Jinwoo Kim Haein Song Jieung Kim 119 1 0 24 Aug 2024
DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Sampling Pala Tej Deep Rishabh Bhardwaj Soujanya Poria MoMe 89 31 0 17 Jun 2024
MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmetic Yuyan Zhou Liang Song Bingning Wang Weipeng Chen MoMe 102 23 0 17 Jun 2024
MindMerger: Efficient Boosting LLM Reasoning in non-English Languages Zixian Huang Wenhao Zhu Gong Cheng Lei Li Fei Yuan LRM 93 14 0 27 May 2024
Jamba: A Hybrid Transformer-Mamba Language Model Opher Lieber Barak Lenz Hofit Bata Gal Cohen Jhonathan Osin ... Nir Ratner N. Rozen Erez Shwartz Mor Zusman Y. Shoham 124 227 0 28 Mar 2024
Checkpoint Merging via Bayesian Optimization in LLM Pretraining Deyuan Liu Zecheng Wang Bingning Wang Weipeng Chen Chunshan Li Zhiying Tu Dianhui Chu Bo Li Dianbo Sui MoMe 97 18 0 28 Mar 2024
The Unreasonable Ineffectiveness of the Deeper Layers Andrey Gromov Kushal Tirumala Hassan Shapourian Paolo Glorioso Daniel A. Roberts 155 106 0 26 Mar 2024
What Makes Quantization for Large Language Models Hard? An Empirical Study from the Lens of Perturbation Zhuocheng Gong Jiahao Liu Jingang Wang Xunliang Cai Dongyan Zhao Rui Yan MQ 57 10 0 11 Mar 2024
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect Xin Men Mingyu Xu Qingyu Zhang Bingning Wang Hongyu Lin Yaojie Lu Xianpei Han Weipeng Chen 117 142 0 06 Mar 2024
Evaluating Quantized Large Language Models Shiyao Li Xuefei Ning Luning Wang Tengxuan Liu Xiangsheng Shi Shengen Yan Guohao Dai Huazhong Yang Yu Wang MQ 119 53 0 28 Feb 2024
SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks Jiwon Song Kyungseok Oh Taesu Kim Hyungjun Kim Yulhwa Kim Jae-Joon Kim 161 34 0 14 Feb 2024
Knowledge Fusion of Large Language Models Fanqi Wan Xinting Huang Deng Cai Xiaojun Quan Wei Bi Shuming Shi MoMe 106 73 0 19 Jan 2024
Mixtral of Experts Albert Q. Jiang Alexandre Sablayrolles Antoine Roux A. Mensch Blanche Savary ... Théophile Gervet Thibaut Lavril Thomas Wang Timothée Lacroix William El Sayed MoE LLMAG 171 1,129 0 08 Jan 2024
Mamba: Linear-Time Sequence Modeling with Selective State Spaces Albert Gu Tri Dao Mamba 168 2,831 0 01 Dec 2023
Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarization Alexandra Chronopoulou Jonas Pfeiffer Joshua Maynez Xinyi Wang Sebastian Ruder Priyanka Agrawal MoMe 87 18 0 15 Nov 2023
Mistral 7B Albert Q. Jiang Alexandre Sablayrolles A. Mensch Chris Bamford Devendra Singh Chaplot ... Teven Le Scao Thibaut Lavril Thomas Wang Timothée Lacroix William El Sayed MoE LRM 157 2,263 0 10 Oct 2023
AdaMerging: Adaptive Model Merging for Multi-Task Learning Enneng Yang Zhenyi Wang Li Shen Shiwei Liu Guibing Guo Xingwei Wang Dacheng Tao MoMe 124 125 0 04 Oct 2023
Deep Model Fusion: A Survey Weishi Li Yong Peng Miao Zhang Liang Ding Han Hu Li Shen FedML MoMe 117 62 0 27 Sep 2023
A Survey on Model Compression for Large Language Models Xunyu Zhu Jian Li Yong Liu Can Ma Weiping Wang 139 233 0 15 Aug 2023
QLoRA: Efficient Finetuning of Quantized LLMs Tim Dettmers Artidoro Pagnoni Ari Holtzman Luke Zettlemoyer ALM 163 2,642 0 23 May 2023
Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models Guillermo Ortiz-Jiménez Alessandro Favero P. Frossard MoMe 163 125 0 22 May 2023
LLM-Pruner: On the Structural Pruning of Large Language Models Xinyin Ma Gongfan Fang Xinchao Wang 175 445 0 19 May 2023
LLaMA: Open and Efficient Foundation Language Models Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux ... Faisal Azhar Aurelien Rodriguez Armand Joulin Edouard Grave Guillaume Lample ALM PILM 1.6K 13,538 0 27 Feb 2023
SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot Elias Frantar Dan Alistarh VLM 133 739 0 02 Jan 2023
Editing Models with Task Arithmetic Gabriel Ilharco Marco Tulio Ribeiro Mitchell Wortsman Suchin Gururangan Ludwig Schmidt Hannaneh Hajishirzi Ali Farhadi KELM MoMe MU 213 522 0 08 Dec 2022
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models Guangxuan Xiao Ji Lin Mickael Seznec Hao Wu Julien Demouth Song Han MQ 257 846 0 18 Nov 2022
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers Elias Frantar Saleh Ashkboos Torsten Hoefler Dan Alistarh MQ 194 1,014 0 31 Oct 2022
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale Tim Dettmers M. Lewis Younes Belkada Luke Zettlemoyer MQ 168 666 0 15 Aug 2022
PaLM: Scaling Language Modeling with Pathways Aakanksha Chowdhery Sharan Narang Jacob Devlin Maarten Bosma Gaurav Mishra ... Kathy Meier-Hellstern Douglas Eck J. Dean Slav Petrov Noah Fiedel PILM LRM 586 6,322 0 05 Apr 2022
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time Mitchell Wortsman Gabriel Ilharco S. Gadre Rebecca Roelofs Raphael Gontijo-Lopes ... Hongseok Namkoong Ali Farhadi Y. Carmon Simon Kornblith Ludwig Schmidt MoMe 201 1,013 1 10 Mar 2022
Measuring Massive Multitask Language Understanding Dan Hendrycks Collin Burns Steven Basart Andy Zou Mantas Mazeika Basel Alomair Jacob Steinhardt ELM RALM 323 4,585 0 07 Sep 2020
Compression of Deep Learning Models for Text: A Survey Manish Gupta Puneet Agrawal VLM MedIm AI4CE 79 119 0 12 Aug 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 1.1K 42,712 0 28 May 2020
Compressing Large-Scale Transformer-Based Models: A Case Study on BERT Prakhar Ganesh Yao Chen Xin Lou Mohammad Ali Khan Yifan Yang Hassan Sajjad Preslav Nakov Deming Chen Marianne Winslett AI4CE 140 201 0 27 Feb 2020
PIQA: Reasoning about Physical Commonsense in Natural Language Yonatan Bisk Rowan Zellers Ronan Le Bras Jianfeng Gao Yejin Choi OOD LRM 311 1,853 0 26 Nov 2019
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions Christopher Clark Kenton Lee Ming-Wei Chang Tom Kwiatkowski Michael Collins Kristina Toutanova 382 1,563 0 24 May 2019
HellaSwag: Can a Machine Really Finish Your Sentence? Rowan Zellers Ari Holtzman Yonatan Bisk Ali Farhadi Yejin Choi 214 2,537 0 19 May 2019
UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction Leland McInnes John Healy James Melville 341 9,526 0 09 Feb 2018
A Survey of Model Compression and Acceleration for Deep Neural Networks Yu Cheng Duo Wang Pan Zhou Zhang Tao 166 1,101 0 23 Oct 2017
RACE: Large-scale ReAding Comprehension Dataset From Examinations Guokun Lai Qizhe Xie Hanxiao Liu Yiming Yang Eduard H. Hovy ELM 304 1,360 0 15 Apr 2017
Pruning Filters for Efficient ConvNets Hao Li Asim Kadav Igor Durdanovic H. Samet H. Graf 3DPC 295 3,709 0 31 Aug 2016
Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding Song Han Huizi Mao W. Dally 3DGS 304 8,876 0 01 Oct 2015
Distilling the Knowledge in a Neural Network Geoffrey E. Hinton Oriol Vinyals J. Dean FedML 369 19,796 0 09 Mar 2015