A Simple and Effective Pruning Approach for Large Language Models

20 June 2023

J. Zico Kolter

Papers citing "A Simple and Effective Pruning Approach for Large Language Models"

23 / 273 papers shown

Title
Pruning Small Pre-Trained Weights Irreversibly and Monotonically Impairs "Difficult" Downstream Tasks in LLMs Lu Yin Ajay Jaiswal Shiwei Liu Souvik Kundu Zhangyang Wang 27 7 0 29 Sep 2023
QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models Yuhui Xu Lingxi Xie Xiaotao Gu Xin Chen Heng Chang Hengheng Zhang Zhensu Chen Xiaopeng Zhang Qi Tian MQ 21 92 0 26 Sep 2023
LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compression Ayush Kaushal Tejas Vaidhya Irina Rish 64 15 0 25 Sep 2023
Pruning Large Language Models via Accuracy Predictor Yupeng Ji Yibo Cao Jiu-si Liu KELM 34 4 0 18 Sep 2023
FPTQ: Fine-grained Post-Training Quantization for Large Language Models Qingyuan Li Yifan Zhang Liang Li Peng Yao Bo-Wen Zhang Xiangxiang Chu Yerui Sun Li-Qiang Du Yuchen Xie MQ 49 12 0 30 Aug 2023
A Survey on Model Compression for Large Language Models Xunyu Zhu Jian Li Yong Liu Can Ma Weiping Wang 36 193 0 15 Aug 2023
H $_2$ O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models Zhenyu Zhang Ying Sheng Dinesh Manocha Tianlong Chen Lianmin Zheng ... Yuandong Tian Christopher Ré Clark W. Barrett Zhangyang Wang Beidi Chen VLM 66 261 0 24 Jun 2023
Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Training A. Nowak Bram Grooten Decebal Constantin Mocanu Jacek Tabor 30 9 0 21 Jun 2023
Winner-Take-All Column Row Sampling for Memory Efficient Adaptation of Language Model Zirui Liu Guanchu Wang Shaochen Zhong Zhaozhuo Xu Daochen Zha ... Zhimeng Jiang Kaixiong Zhou V. Chaudhary Shuai Xu Xia Hu 47 12 0 24 May 2023
CrAFT: Compression-Aware Fine-Tuning for Efficient Visual Task Adaptation J. Heo S. Azizi A. Fayyazi Massoud Pedram 28 0 0 08 May 2023
Sparks of Artificial General Intelligence: Early experiments with GPT-4 Sébastien Bubeck Varun Chandrasekaran Ronen Eldan J. Gehrke Eric Horvitz ... Scott M. Lundberg Harsha Nori Hamid Palangi Marco Tulio Ribeiro Yi Zhang ELM AI4MH AI4CE ALM 360 3,029 0 22 Mar 2023
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU Ying Sheng Lianmin Zheng Binhang Yuan Zhuohan Li Max Ryabinin ... Joseph E. Gonzalez Percy Liang Christopher Ré Ion Stoica Ce Zhang 149 371 0 13 Mar 2023
DepGraph: Towards Any Structural Pruning Gongfan Fang Xinyin Ma Mingli Song Michael Bi Mi Xinchao Wang GNN 91 258 0 30 Jan 2023
Outliers Dimensions that Disrupt Transformers Are Driven by Frequency Giovanni Puccetti Anna Rogers Aleksandr Drozd F. Dell’Orletta 81 42 0 23 May 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 416 8,650 0 28 Jan 2022
All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality William Timkey Marten van Schijndel 226 111 0 09 Sep 2021
An Information Theory-inspired Strategy for Automatic Network Pruning Xiawu Zheng Yuexiao Ma Teng Xi Gang Zhang Errui Ding Yuchao Li Jie Chen Yonghong Tian Rongrong Ji 48 13 0 19 Aug 2021
Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks Itay Hubara Brian Chmiel Moshe Island Ron Banner S. Naor Daniel Soudry 59 111 0 16 Feb 2021
Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks Torsten Hoefler Dan Alistarh Tal Ben-Nun Nikoli Dryden Alexandra Peste MQ 144 685 0 31 Jan 2021
The Lottery Ticket Hypothesis for Pre-trained BERT Networks Tianlong Chen Jonathan Frankle Shiyu Chang Sijia Liu Yang Zhang Zhangyang Wang Michael Carbin 156 345 0 23 Jul 2020
What is the State of Neural Network Pruning? Davis W. Blalock Jose Javier Gonzalez Ortiz Jonathan Frankle John Guttag 191 1,032 0 06 Mar 2020
Comparing Rewinding and Fine-tuning in Neural Network Pruning Alex Renda Jonathan Frankle Michael Carbin 235 383 0 05 Mar 2020
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 299 6,996 0 20 Apr 2018