LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning

26 March 2024

Tong Zhang

Papers citing "LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning"

36 / 36 papers shown

Title
Reinforcement Learning Finetunes Small Subnetworks in Large Language Models Sagnik Mukherjee Lifan Yuan Dilek Hakkani-Tur Hao Peng 7 0 0 16 May 2025
Sparsity Outperforms Low-Rank Projections in Few-Shot Adaptation Nairouz Mrabah Nicolas Richet Ismail Ben Ayed Eric Granger BDL VLM 58 0 0 16 Apr 2025
Steering off Course: Reliability Challenges in Steering Language Models Patrick Queiroz Da Silva Hari Sethuraman Dheeraj Rajagopal Hannaneh Hajishirzi Sachin Kumar LLMSV 31 1 0 06 Apr 2025
ClusComp: A Simple Paradigm for Model Compression and Efficient Finetuning Baohao Liao Christian Herold Seyyed Hadi Hashemi Stefan Vasilev Shahram Khadivi Christof Monz MQ 44 0 0 17 Mar 2025
Quantum-PEFT: Ultra parameter-efficient fine-tuning Toshiaki Koike-Akino F. Tonin Yongtao Wu Frank Zhengqing Wu Leyla Naz Candogan V. Cevher MQ 54 3 0 07 Mar 2025
NEAT: Nonlinear Parameter-efficient Adaptation of Pre-trained Models Yibo Zhong Haoxiang Jiang Lincan Li Ryumei Nakada Tianci Liu Linjun Zhang Huaxiu Yao Haoyu Wang 79 2 0 24 Feb 2025
COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMs Liming Liu Zhenghao Xu Zixuan Zhang Hao Kang Zichong Li Chen Liang Weizhu Chen T. Zhao 146 1 0 24 Feb 2025
Enhancing Adversarial Robustness of Vision-Language Models through Low-Rank Adaptation Yuheng Ji Yue Liu Zhicheng Zhang Zhao Zhang Yuting Zhao Gang Zhou Xingwei Zhang Xinwang Liu Xiaolong Zheng VLM 115 4 0 21 Feb 2025
CE-LoRA: Computation-Efficient LoRA Fine-Tuning for Language Models Guanduo Chen Yutong He Yipeng Hu Kun Yuan Binhang Yuan 54 0 0 03 Feb 2025
SubTrack your Grad: Gradient Subspace Tracking for Memory and Time Efficient Full-Parameter LLM Training Sahar Rajabi Nayeema Nonta Sirisha Rambhatla 90 0 0 03 Feb 2025
Activating Distributed Visual Region within LLMs for Efficient and Effective Vision-Language Training and Inference Siyuan Wang Dianyi Wang Chengxing Zhou Zejun Li Zhihao Fan Xuanjing Huang Zhongyu Wei VLM 209 0 0 17 Dec 2024
Enhancing the Reasoning Capabilities of Small Language Models via Solution Guidance Fine-Tuning Jing Bi Yuting Wu Weiwei Xing Zhenjie Wei ReLM OffRL LRM 88 4 0 13 Dec 2024
FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Training Philip Zmushko Aleksandr Beznosikov Martin Takáč Samuel Horváth 44 0 0 12 Nov 2024
Understanding Layer Significance in LLM Alignment Guangyuan Shi Zexin Lu Xiaoyu Dong Wenlong Zhang Xuanyu Zhang Yujie Feng Xiao-Ming Wu 58 2 0 23 Oct 2024
Layer-wise Importance Matters: Less Memory for Better Performance in Parameter-efficient Fine-tuning of Large Language Models Kai Yao P. Gao Lichun Li Yuan Zhao Xiaofeng Wang Wei Wang Jianke Zhu 26 1 0 15 Oct 2024
Simultaneous Computation and Memory Efficient Zeroth-Order Optimizer for Fine-Tuning Large Language Models Fei Wang Li Shen Liang Ding Chao Xue Ye Liu Changxing Ding 32 0 0 13 Oct 2024
Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbation Guozhi Liu Weiwei Lin Tiansheng Huang Ruichao Mo Qi Mu Li Shen AAML 66 10 0 13 Oct 2024
Multitask Mayhem: Unveiling and Mitigating Safety Gaps in LLMs Fine-tuning Essa Jan Nouar Aldahoul Moiz Ali Faizan Ahmad Fareed Zaffar Yasir Zaki 34 3 0 18 Sep 2024
An overview of domain-specific foundation model: key technologies, applications and challenges Haolong Chen Hanzhi Chen Zijian Zhao Kaifeng Han Guangxu Zhu Yichen Zhao Ying Du Wei Xu Qingjiang Shi ALM VLM 64 4 0 06 Sep 2024
SORSA: Singular Values and Orthonormal Regularized Singular Vectors Adaptation of Large Language Models Yang Cao 44 2 0 21 Aug 2024
Flexora: Flexible Low Rank Adaptation for Large Language Models Chenxing Wei Yao Shu Ying Tiffany He Fei Richard Yu AI4CE 34 3 0 20 Aug 2024
Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients Zhenyu (Allen) Zhang Ajay Jaiswal L. Yin Shiwei Liu Jiawei Zhao Yuandong Tian Zhangyang Wang VLM 33 16 0 11 Jul 2024
A Survey on LoRA of Large Language Models Yuren Mao Yuhang Ge Yijiang Fan Wenyi Xu Yu Mi Zhonghao Hu Yunjun Gao ALM 54 25 0 08 Jul 2024
LoRA-GA: Low-Rank Adaptation with Gradient Approximation Shaowen Wang Linxi Yu Jian Li ALM AI4CE 26 29 0 06 Jul 2024
ScaleBiO: Scalable Bilevel Optimization for LLM Data Reweighting Rui Pan Jipeng Zhang Xingyuan Pan Renjie Pi Xiaoyu Wang Tong Zhang 58 5 0 28 Jun 2024
Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradients Aashiq Muhamed Oscar Li David Woodruff Mona Diab Virginia Smith 61 7 0 25 Jun 2024
RankAdaptor: Hierarchical Dynamic Low-Rank Adaptation for Structural Pruned LLMs Changhai Zhou Shijie Han Shiyang Zhang Shichao Weng Zekai Liu Cheng Jin 45 1 0 22 Jun 2024
LaMDA: Large Model Fine-Tuning via Spectrally Decomposed Low-Dimensional Adaptation Seyedarmin Azizi Souvik Kundu Massoud Pedram 32 7 0 18 Jun 2024
MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning Hanqing Wang Zeguan Xiao Shuo Wang Guanhua Chen Guanhua Chen 44 19 0 13 Jun 2024
OwLore: Outlier-weighed Layerwise Sampled Low-Rank Projection for Memory-Efficient LLM Fine-tuning Pengxiang Li Lu Yin Xiaowei Gao Shiwei Liu 34 7 0 28 May 2024
BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Models Qi Luo Hengxu Yu Xiao Li 47 1 0 03 Apr 2024
SmartFRZ: An Efficient Training Framework using Attention-Based Layer Freezing Sheng Li Geng Yuan Yuezhen Dai Youtao Zhang Yanzhi Wang Xulong Tang 31 18 0 30 Jan 2024
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 339 12,003 0 04 Mar 2022
ZeRO-Offload: Democratizing Billion-Scale Model Training Jie Ren Samyam Rajbhandari Reza Yazdani Aminabadi Olatunji Ruwase Shuangyang Yang Minjia Zhang Dong Li Yuxiong He MoE 177 416 0 18 Jan 2021
WARP: Word-level Adversarial ReProgramming Karen Hambardzumyan Hrant Khachatrian Jonathan May AAML 254 342 0 01 Jan 2021
PubMedQA: A Dataset for Biomedical Research Question Answering Qiao Jin Bhuwan Dhingra Zhengping Liu William W. Cohen Xinghua Lu 237 815 0 13 Sep 2019