Make Pre-trained Model Reversible: From Parameter to Memory Efficient Fine-Tuning

1 June 2023

Baohao Liao

Shaomu Tan

Christof Monz

KELM

ArXiv PDF HTML

Papers citing "Make Pre-trained Model Reversible: From Parameter to Memory Efficient Fine-Tuning"

27 / 27 papers shown

Title
Enhancing Ultra-Low-Bit Quantization of Large Language Models Through Saliency-Aware Partial Retraining Deyu Cao Samin Aref MQ 48 0 0 14 Apr 2025
Generative Artificial Intelligence for Internet of Things Computing: A Systematic Survey Fabrizio Mangione Claudio Savaglio Giancarlo Fortino 44 0 0 10 Apr 2025
MobiLLM: Enabling LLM Fine-Tuning on the Mobile Device via Server Assisted Side Tuning Liang Li Xingke Yang Wen Wu Hao Wang Tomoaki Ohtsuki Xin Fu Miao Pan Xuemin Shen 49 1 0 27 Feb 2025
DP-MemArc: Differential Privacy Transfer Learning for Memory Efficient Language Models Yanming Liu Xinyue Peng Yuwei Zhang Xiaolan Ke Songhang Deng ... Sheng Cheng Xun Wang Yuxiang Cai Tianyu Du Xuhong Zhang 118 0 0 21 Feb 2025
SSH: Sparse Spectrum Adaptation via Discrete Hartley Transformation Yixian Shen Qi Bi Jia-Hong Huang Hongyi Zhu Andy D. Pimentel Anuj Pathania 69 0 0 08 Feb 2025
Memory-Efficient Fine-Tuning of Transformers via Token Selection Antoine Simoulin Namyong Park Xiaoyi Liu Grey Yang 128 0 0 31 Jan 2025
Activating Distributed Visual Region within LLMs for Efficient and Effective Vision-Language Training and Inference Siyuan Wang Dianyi Wang Chengxing Zhou Zejun Li Zhihao Fan Xuanjing Huang Zhongyu Wei VLM 323 0 0 17 Dec 2024
Parameter-Efficient Fine-Tuning via Selective Discrete Cosine Transform Yixian Shen Qi Bi Jia-Hong Huang Hongyi Zhu Anuj Pathania 44 1 0 09 Oct 2024
LoRTA: Low Rank Tensor Adaptation of Large Language Models Ignacio Hounie Charilaos I. Kanatsoulis Arnuv Tandon Alejandro Ribeiro 65 0 0 05 Oct 2024
CoRA: Optimizing Low-Rank Adaptation with Common Subspace of Large Language Models Xiaojun Xiao Sen Shen Qiming Bao Hongfei Rong Kairui Liu Zhongsheng Wang Jing Liu 51 2 0 31 Aug 2024
Pre-training Everywhere: Parameter-Efficient Fine-Tuning for Medical Image Analysis via Target Parameter Pre-training Xingliang Lei Yiwen Ye Ziyang Chen Minglei Shu Yong Xia 55 1 0 27 Aug 2024
Robust Adaptation of Foundation Models with Black-Box Visual Prompting Changdae Oh Gyeongdeok Seo Geunyoung Jung Zhi-Qi Cheng Hosik Choi Jiyoung Jung Kyungwoo Song VLM 57 1 0 04 Jul 2024
MEFT: Memory-Efficient Fine-Tuning through Sparse Adapter Jitai Hao Weiwei Sun Xin Xin Qi Meng Zhumin Chen Pengjie Ren Zhaochun Ren MoE 47 2 0 07 Jun 2024
Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruning Naibin Gu Peng Fu Xiyu Liu Bowen Shen Zheng Lin Weiping Wang 50 6 0 06 Jun 2024
Neuron Specialization: Leveraging intrinsic task modularity for multilingual machine translation Shaomu Tan Di Wu Christof Monz MoMe 50 8 0 17 Apr 2024
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey Zeyu Han Chao Gao Jinyang Liu Jeff Zhang Sai Qian Zhang 150 336 0 21 Mar 2024
STAR: Constraint LoRA with Dynamic Active Learning for Data-Efficient Fine-Tuning of Large Language Models Linhai Zhang Jialong Wu Deyu Zhou Guoqiang Xu 44 4 0 02 Mar 2024
ApiQ: Finetuning of 2-Bit Quantized Large Language Model Baohao Liao Christian Herold Shahram Khadivi Christof Monz CLL MQ 67 13 0 07 Feb 2024
Towards Green AI in Fine-tuning Large Language Models via Adaptive Backpropagation Kai Huang Hanyu Yin Heng Huang Wei Gao 38 11 0 22 Sep 2023
PaReprop: Fast Parallelized Reversible Backpropagation Tyler Lixuan Zhu K. Mangalam 36 1 0 15 Jun 2023
AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition Shoufa Chen Chongjian Ge Zhan Tong Jiangliu Wang Yibing Song Jue Wang Ping Luo 154 647 0 26 May 2022
Masked Autoencoders Are Scalable Vision Learners Kaiming He Xinlei Chen Saining Xie Yanghao Li Piotr Dollár Ross B. Girshick ViT TPM 327 7,544 0 11 Nov 2021
The Power of Scale for Parameter-Efficient Prompt Tuning Brian Lester Rami Al-Rfou Noah Constant VPVLM 298 3,906 0 18 Apr 2021
ZeRO-Offload: Democratizing Billion-Scale Model Training Jie Ren Samyam Rajbhandari Reza Yazdani Aminabadi Olatunji Ruwase Shuangyang Yang Minjia Zhang Dong Li Yuxiong He MoE 184 419 0 18 Jan 2021
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 270 4,576 0 23 Jan 2020
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 316 7,044 0 20 Apr 2018
Neural Architecture Search with Reinforcement Learning Barret Zoph Quoc V. Le 278 5,336 0 05 Nov 2016