BitDelta: Your Fine-Tune May Only Be Worth One Bit

BitDelta: Your Fine-Tune May Only Be Worth One Bit

15 February 2024

Song Han

Tianle Cai

Papers citing "BitDelta: Your Fine-Tune May Only Be Worth One Bit"

18 / 18 papers shown

Title
ImPart: Importance-Aware Delta-Sparsification for Improved Model Compression and Merging in LLMs Yan Yang Yixia Li Hongru Wang Xuetao Wei Jianqiao Yu Yun-Nung Chen Guanhua Chen MoMe 30 0 0 17 Apr 2025
Seeing Delta Parameters as JPEG Images: Data-Free Delta Compression with Discrete Cosine Transform Chenyu Huang Peng Ye Xueliang Wang Shenghe Zheng Biqing Qi Lei Bai Wanli Ouyang Tao Chen 36 0 0 09 Mar 2025
Delta Decompression for MoE-based LLMs Compression Hao Gu Wei Li Lujun Li Qiyuan Zhu Mark Lee Shengjie Sun Wei Xue Yike Guo MoE 54 0 0 24 Feb 2025
1bit-Merging: Dynamic Quantized Merging for Large Language Models Shuqi Liu Han Wu Bowei He Zehua Liu Xiongwei Han M. Yuan Linqi Song MoMe MQ 81 2 0 15 Feb 2025
Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusion Zhuokun Chen Jinwu Hu Zeshuai Deng Yufeng Wang Bohan Zhuang Mingkui Tan 71 0 0 02 Dec 2024
Quantized Delta Weight Is Safety Keeper Yule Liu Zhen Sun Xinlei He Xinyi Huang 96 2 0 29 Nov 2024
Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologies Liwen Wang Sheng Chen Linnan Jiang Shu Pan Runze Cai Sen Yang Fei Yang 52 3 0 24 Oct 2024
A Unified View of Delta Parameter Editing in Post-Trained Large-Scale Models Qiaoyu Tang Le Yu Bowen Yu Hongyu Lin Keming Lu Yaojie Lu Xianpei Han Le Sun MoMe 39 1 0 17 Oct 2024
DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Models Wenlong Deng Yize Zhao V. Vakilian Minghui Chen Xiaoxiao Li Christos Thrampoulidis 47 3 0 12 Oct 2024
DeltaDQ: Ultra-High Delta Compression for Fine-Tuned LLMs via Group-wise Dropout and Separate Quantization Yanfeng Jiang Zelan Yang B. Chen Shen Li Yong Li Tao Li MQ 36 0 0 11 Oct 2024
Novel-WD: Exploring acquisition of Novel World Knowledge in LLMs Using Prefix-Tuning Maxime Méloux Christophe Cerisara KELM CLL 32 0 0 30 Aug 2024
Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMs Ashwinee Panda Berivan Isik Xiangyu Qi Sanmi Koyejo Tsachy Weissman Prateek Mittal MoMe 45 15 0 24 Jun 2024
ME-Switch: A Memory-Efficient Expert Switching Framework for Large Language Models Jing Liu Ruihao Gong Mingyang Zhang Yefei He Jianfei Cai Bohan Zhuang MoE 45 0 0 13 Jun 2024
Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Models Bowen Ping Shuo Wang Hanqing Wang Xu Han Yuzhuang Xu Yukun Yan Yun Chen Baobao Chang Zhiyuan Liu Maosong Sun MQ 51 5 0 13 Jun 2024
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey Zeyu Han Chao Gao Jinyang Liu Jeff Zhang Sai Qian Zhang 150 319 0 21 Mar 2024
ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization Prateek Yadav Leshem Choshen Colin Raffel Mohit Bansal 32 13 0 22 Nov 2023
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 375 12,081 0 04 Mar 2022
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press Noah A. Smith M. Lewis 253 701 0 27 Aug 2021