Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs

13 October 2023

Papers citing "Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs"

34 / 34 papers shown

Title
Accelerating LLM Inference with Flexible N:M Sparsity via A Fully Digital Compute-in-Memory Accelerator Akshat Ramachandran Souvik Kundu Arnab Raha Shamik Kundu Deepak K. Mathaikutty Tushar Krishna 32 1 0 19 Apr 2025
Model Hemorrhage and the Robustness Limits of Large Language Models Ziyang Ma Zehan Li L. Zhang Gui-Song Xia Bo Du Liangpei Zhang Dacheng Tao 59 0 0 31 Mar 2025
EvoP: Robust LLM Inference via Evolutionary Pruning Shangyu Wu Hongchao Du Ying Xiong Shuai Chen Tei-Wei Kuo Nan Guan Chun Jason Xue 34 1 0 19 Feb 2025
Symmetric Pruning of Large Language Models Kai Yi Peter Richtárik AAML VLM 62 0 0 31 Jan 2025
DynST: Dynamic Sparse Training for Resource-Constrained Spatio-Temporal Forecasting Hao Wu Haomin Wen Guibin Zhang Yutong Xia Kai Wang Yuxuan Liang Yu Zheng Kun Wang 68 2 0 17 Jan 2025
Zeroth-Order Adaptive Neuron Alignment Based Pruning without Re-Training Elia Cunegatti Leonardo Lucio Custode Giovanni Iacca 49 0 0 11 Nov 2024
Beware of Calibration Data for Pruning Large Language Models Yixin Ji Yang Xiang Juntao Li Qingrong Xia Ping Li Xinyu Duan Zhefeng Wang Min Zhang 42 2 0 23 Oct 2024
Is C4 Dataset Optimal for Pruning? An Investigation of Calibration Data for LLM Pruning Abhinav Bandari L. Yin Cheng-Yu Hsieh Ajay Kumar Jaiswal Tianlong Chen Li Shen Ranjay Krishna Shiwei Liu 36 6 0 09 Oct 2024
OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decomposition Stephen Zhang V. Papyan VLM 51 1 0 20 Sep 2024
LLM-Barber: Block-Aware Rebuilder for Sparsity Mask in One-Shot for Large Language Models Yupeng Su Ziyi Guan Xiaoqun Liu Tianlai Jin Dongkuan Wu G. Chesi Ngai Wong Hao Yu 40 1 0 20 Aug 2024
Enhancing One-shot Pruned Pre-trained Language Models through Sparse-Dense-Sparse Mechanism Guanchen Li Xiandong Zhao Lian Liu Zeping Li Dong Li Lu Tian Jie He Ashish Sirasao E. Barsoum VLM 32 0 0 20 Aug 2024
FactorLLM: Factorizing Knowledge via Mixture of Experts for Large Language Models Zhongyu Zhao Menghang Dong Rongyu Zhang Wenzhao Zheng Yunpeng Zhang Huanrui Yang Dalong Du Kurt Keutzer Shanghang Zhang 48 0 0 15 Aug 2024
Pruning Large Language Models with Semi-Structural Adaptive Sparse Training Weiyu Huang Yuezhou Hu Guohao Jian Jun Zhu Jianfei Chen 35 5 0 30 Jul 2024
Rethinking Pruning Large Language Models: Benefits and Pitfalls of Reconstruction Error Minimization Sungbin Shin Wonpyo Park Jaeho Lee Namhoon Lee 38 1 0 21 Jun 2024
DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Sampling Pala Tej Deep Rishabh Bhardwaj Soujanya Poria MoMe 32 24 0 17 Jun 2024
ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Models Xiang Meng Kayhan Behdin Haoyue Wang Rahul Mazumder 42 3 0 12 Jun 2024
SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretraining Andi Han Jiaxiang Li Wei Huang Mingyi Hong Akiko Takeda Pratik Jawanpuria Bamdev Mishra 44 10 0 04 Jun 2024
SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models Xudong Lu Aojun Zhou Yuhui Xu Renrui Zhang Peng Gao Hongsheng Li 31 7 0 25 May 2024
A Survey on Efficient Inference for Large Language Models Zixuan Zhou Xuefei Ning Ke Hong Tianyu Fu Jiaming Xu ... Shengen Yan Guohao Dai Xiao-Ping Zhang Yuhan Dong Yu-Xiang Wang 46 83 0 22 Apr 2024
Rethinking Pruning for Vision-Language Models: Strategies for Effective Sparsity and Performance Restoration Shwai He Ang Li Tianlong Chen VLM 44 1 0 03 Apr 2024
AffineQuant: Affine Transformation Quantization for Large Language Models Yuexiao Ma Huixia Li Xiawu Zheng Feng Ling Xuefeng Xiao Rui Wang Shilei Wen Rongrong Ji Rongrong Ji MQ 40 19 0 19 Mar 2024
DPPA: Pruning Method for Large Language Model to Model Merging Yaochen Zhu Rui Xia Jiajun Zhang MoMe 35 4 0 05 Mar 2024
Model Compression and Efficient Inference for Large Language Models: A Survey Wenxiao Wang Wei Chen Yicong Luo Yongliu Long Zhengkai Lin Liye Zhang Binbin Lin Deng Cai Xiaofei He MQ 41 48 0 15 Feb 2024
SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks Jiwon Song Kyungseok Oh Taesu Kim Hyungjun Kim Yulhwa Kim Jae-Joon Kim 68 20 0 14 Feb 2024
Two Heads Are Better Than One: Boosting Graph Sparse Training via Semantic and Topological Awareness Guibin Zhang Xinfeng Li Kun Wang Fan Zhang Yongduo Sui Kai Wang Yuxuan Liang Dawei Cheng Shirui Pan Tianlong Chen 27 10 0 02 Feb 2024
Fast and Optimal Weight Update for Pruned Large Language Models Vladimír Boza 27 6 0 01 Jan 2024
PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMs Max Zimmer Megi Andoni Christoph Spiegel Sebastian Pokutta VLM 52 10 0 23 Dec 2023
The Efficiency Spectrum of Large Language Models: An Algorithmic Survey Tianyu Ding Tianyi Chen Haidong Zhu Jiachen Jiang Yiqi Zhong Jinxin Zhou Guangzhi Wang Zhihui Zhu Ilya Zharkov Luming Liang 27 22 0 01 Dec 2023
Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity Lu Yin You Wu Zhenyu (Allen) Zhang Cheng-Yu Hsieh Yaqing Wang ... Mykola Pechenizkiy Yi Liang Michael Bendersky Zhangyang Wang Shiwei Liu 28 78 0 08 Oct 2023
OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models Wenqi Shao Mengzhao Chen Zhaoyang Zhang Peng-Tao Xu Lirui Zhao Zhiqiang Li Kaipeng Zhang Peng Gao Yu Qiao Ping Luo MQ 15 176 0 25 Aug 2023
A Survey on Model Compression for Large Language Models Xunyu Zhu Jian Li Yong Liu Can Ma Weiping Wang 36 193 0 15 Aug 2023
Sparks of Artificial General Intelligence: Early experiments with GPT-4 Sébastien Bubeck Varun Chandrasekaran Ronen Eldan J. Gehrke Eric Horvitz ... Scott M. Lundberg Harsha Nori Hamid Palangi Marco Tulio Ribeiro Yi Zhang ELM AI4MH AI4CE ALM 298 2,232 0 22 Mar 2023
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 385 8,495 0 28 Jan 2022
Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks Itay Hubara Brian Chmiel Moshe Island Ron Banner S. Naor Daniel Soudry 53 111 0 16 Feb 2021