Efficient LLMs with AMP: Attention Heads and MLP Pruning

Efficient LLMs with AMP: Attention Heads and MLP Pruning

29 April 2025

Leandro Giusti Mugnaini

Lucas Lauton de Alcantara

Victor Zacarias

ArXiv (abs)PDF HTML

Papers citing "Efficient LLMs with AMP: Attention Heads and MLP Pruning"

17 / 17 papers shown

Title
Holistically Evaluating the Environmental Impact of Creating Language Models Jacob Morrison Clara Na Jared Fernandez Tim Dettmers Emma Strubell Jesse Dodge ELM 76 3 0 03 Mar 2025
You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learning Ayan Sengupta Siddhant Chaudhary Tanmoy Chakraborty 88 4 0 25 Jan 2025
The Unreasonable Ineffectiveness of the Deeper Layers Andrey Gromov Kushal Tirumala Hassan Shapourian Paolo Glorioso Daniel A. Roberts 121 99 0 26 Mar 2024
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey Zeyu Han Chao Gao Jinyang Liu Jeff Zhang Sai Qian Zhang 233 395 0 21 Mar 2024
The LLM Surgeon Tycho F. A. van der Ouderaa Markus Nagel M. V. Baalen Yuki Markus Asano Tijmen Blankevoort 57 16 0 28 Dec 2023
Fluctuation-based Adaptive Structured Pruning for Large Language Models Yongqi An Xu Zhao Tao Yu Ming Tang Jinqiao Wang 99 55 0 19 Dec 2023
Revisiting Random Channel Pruning for Neural Network Compression Yawei Li Kamil Adamczewski Wen Li Shuhang Gu Radu Timofte Luc Van Gool 88 86 0 11 May 2022
Training Compute-Optimal Large Language Models Jordan Hoffmann Sebastian Borgeaud A. Mensch Elena Buchatskaya Trevor Cai ... Karen Simonyan Erich Elsen Jack W. Rae Oriol Vinyals Laurent Sifre AI4TS 203 1,949 0 29 Mar 2022
The Efficiency Misnomer Daoyuan Chen Liuyi Yao Dawei Gao Ashish Vaswani Yaliang Li 99 103 0 25 Oct 2021
LoRA: Low-Rank Adaptation of Large Language Models J. E. Hu Yelong Shen Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang Weizhu Chen OffRL AI4TS AI4CE ALM AIMat 474 10,367 0 17 Jun 2021
Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch Aojun Zhou Yukun Ma Junnan Zhu Jianbo Liu Zhijie Zhang Kun Yuan Wenxiu Sun Hongsheng Li 201 247 0 08 Feb 2021
Neural Pruning via Growing Regularization Huan Wang Can Qin Yulun Zhang Y. Fu 83 145 0 16 Dec 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 795 42,055 0 28 May 2020
Quantifying the Carbon Emissions of Machine Learning Alexandre Lacoste A. Luccioni Victor Schmidt Thomas Dandres 94 707 0 21 Oct 2019
Green AI Roy Schwartz Jesse Dodge Noah A. Smith Oren Etzioni 107 1,144 0 22 Jul 2019
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 701 131,652 0 12 Jun 2017
Pointer Sentinel Mixture Models Stephen Merity Caiming Xiong James Bradbury R. Socher RALM 328 2,876 0 26 Sep 2016