DeltaDQ: Ultra-High Delta Compression for Fine-Tuned LLMs via Group-wise
Dropout and Separate Quantization

DeltaDQ: Ultra-High Delta Compression for Fine-Tuned LLMs via Group-wise Dropout and Separate Quantization

11 October 2024

Zelan Yang

ArXiv (abs)PDF HTML

Papers citing "DeltaDQ: Ultra-High Delta Compression for Fine-Tuned LLMs via Group-wise Dropout and Separate Quantization"

16 / 16 papers shown

Title
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey Zeyu Han Chao Gao Jinyang Liu Jeff Zhang Sai Qian Zhang 259 398 0 21 Mar 2024
Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch Le Yu Yu Bowen Haiyang Yu Fei Huang Yongbin Li MoMe 111 335 0 06 Nov 2023
Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity Lu Yin You Wu Zhenyu Zhang Cheng-Yu Hsieh Yaqing Wang ... Mykola Pechenizkiy Yi Liang Michael Bendersky Zhangyang Wang Shiwei Liu 131 102 0 08 Oct 2023
WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct Haipeng Luo Qingfeng Sun Can Xu Pu Zhao Jian-Guang Lou ... Xiubo Geng Qingwei Lin Shifeng Chen Yansong Tang Dongmei Zhang LRM OSLM 267 467 0 18 Aug 2023
WizardCoder: Empowering Code Large Language Models with Evol-Instruct Ziyang Luo Can Xu Pu Zhao Qingfeng Sun Xiubo Geng Wenxiang Hu Chongyang Tao Jing Ma Qingwei Lin Daxin Jiang ELM SyDa ALM 138 692 0 14 Jun 2023
GPT-4 Technical Report OpenAI OpenAI OpenAI Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad ... Shengjia Zhao Tianhao Zheng Juntang Zhuang William Zhuk Barret Zoph LLMAG MLLM 1.5K 14,761 0 15 Mar 2023
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers Elias Frantar Saleh Ashkboos Torsten Hoefler Dan Alistarh MQ 155 1,008 0 31 Oct 2022
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale Tim Dettmers M. Lewis Younes Belkada Luke Zettlemoyer MQ 119 664 0 15 Aug 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 897 13,228 0 04 Mar 2022
Revisiting Parameter-Efficient Tuning: Are We Really There Yet? Guanzheng Chen Fangyu Liu Zaiqiao Meng Shangsong Liang 64 94 0 16 Feb 2022
Training Verifiers to Solve Math Word Problems K. Cobbe V. Kosaraju Mohammad Bavarian Mark Chen Heewoo Jun ... Jerry Tworek Jacob Hilton Reiichiro Nakano Christopher Hesse John Schulman ReLM OffRL LRM 367 4,598 0 27 Oct 2021
Evaluating Large Language Models Trained on Code Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Pondé ... Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever Wojciech Zaremba ELM ALM 238 5,675 0 07 Jul 2021
LoRA: Low-Rank Adaptation of Large Language Models J. E. Hu Yelong Shen Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang Weizhu Chen OffRL AI4TS AI4CE ALM AIMat 504 10,526 0 17 Jun 2021
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 905 42,520 0 28 May 2020
HAQ: Hardware-Aware Automated Quantization with Mixed Precision Kuan-Chieh Wang Zhijian Liu Chengyue Wu Ji Lin Song Han MQ 129 884 0 21 Nov 2018
Learning both Weights and Connections for Efficient Neural Networks Song Han Jeff Pool J. Tran W. Dally CVBM 320 6,715 0 08 Jun 2015