LoRA-FA: Memory-efficient Low-rank Adaptation for Large Language Models Fine-tuning

7 August 2023

Papers citing "LoRA-FA: Memory-efficient Low-rank Adaptation for Large Language Models Fine-tuning"

21 / 21 papers shown

Title
A Hessian-informed hyperparameter optimization for differential learning rate Shiyun Xu Zhiqi Bu Yiliang Zhang Ian J. Barnett 39 1 0 20 May 2025
Parameter-Efficient Fine-Tuning of Large Language Models via Deconvolution in Subspace Jia-Chen Zhang Yu-Jie Xiong Chun-Ming Xia Dong-Hai Zhu Xi-He Qiu 64 1 0 03 Mar 2025
K-LoRA: Unlocking Training-Free Fusion of Any Subject and Style LoRAs Ziheng Ouyang Zhen Li Qibin Hou MoMe OffRL 133 2 0 25 Feb 2025
GoRA: Gradient-driven Adaptive Low Rank Adaptation Haonan He Peng Ye Yuchen Ren Yuan Yuan Lei Chen AI4TS AI4CE 169 0 0 13 Feb 2025
LoCA: Location-Aware Cosine Adaptation for Parameter-Efficient Fine-Tuning Zhekai Du Yinjie Min Jingjing Li Ke Lu Changliang Zou Liuhua Peng Tingjin Chu M. Gong 159 1 0 05 Feb 2025
Memory-Efficient Fine-Tuning of Transformers via Token Selection Antoine Simoulin Namyong Park Xiaoyi Liu Grey Yang 110 0 0 31 Jan 2025
Sparse High Rank Adapters K. Bhardwaj N. Pandey Sweta Priyadarshi Viswanath Ganapathy Rafael Esteves ... P. Whatmough Risheek Garrepalli M. V. Baalen Harris Teague Markus Nagel MQ 43 4 0 28 Jan 2025
Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologies L. Wang Sheng Chen Linnan Jiang Shu Pan Runze Cai Sen Yang Fei Yang 49 3 0 24 Oct 2024
LoRTA: Low Rank Tensor Adaptation of Large Language Models Ignacio Hounie Charilaos I. Kanatsoulis Arnuv Tandon Alejandro Ribeiro 36 0 0 05 Oct 2024
Selective Aggregation for Low-Rank Adaptation in Federated Learning Pengxin Guo Shuang Zeng Y. Wang Huijie Fan Feifei Wang Liangqiong Qu FedML 44 8 0 02 Oct 2024
CADC: Encoding User-Item Interactions for Compressing Recommendation Model Training Data Hossein Entezari Zarch Abdulla Alshabanah Chaoyi Jiang Murali Annavaram 23 1 0 11 Jul 2024
ShareLoRA: Parameter Efficient and Robust Large Language Model Fine-tuning via Shared Low-Rank Adaptation Yurun Song Junchen Zhao Ian G. Harris S. Jyothi 32 3 0 16 Jun 2024
The Impact of Initialization on LoRA Finetuning Dynamics Soufiane Hayou Nikhil Ghosh Bin Yu AI4CE 36 10 0 12 Jun 2024
LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking Jialin Li Qiang Nie Weifu Fu Yuhuan Lin Guangpin Tao Yong-Jin Liu Chengjie Wang 32 4 0 07 Mar 2024
GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection Jiawei Zhao Zhenyu (Allen) Zhang Beidi Chen Zhangyang Wang A. Anandkumar Yuandong Tian 43 173 0 06 Mar 2024
Run LoRA Run: Faster and Lighter LoRA Implementations Daria Cherniuk A. Mikhalev Ivan V. Oseledets AI4CE 22 1 0 06 Dec 2023
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 313 11,953 0 04 Mar 2022
The Power of Scale for Parameter-Efficient Prompt Tuning Brian Lester Rami Al-Rfou Noah Constant VPVLM 280 3,848 0 18 Apr 2021
ZeRO-Offload: Democratizing Billion-Scale Model Training Jie Ren Samyam Rajbhandari Reza Yazdani Aminabadi Olatunji Ruwase Shuangyang Yang Minjia Zhang Dong Li Yuxiong He MoE 168 414 0 18 Jan 2021
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 245 1,821 0 17 Sep 2019
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 297 6,956 0 20 Apr 2018