LoRA-GA: Low-Rank Adaptation with Gradient Approximation

6 July 2024

Papers citing "LoRA-GA: Low-Rank Adaptation with Gradient Approximation"

25 / 25 papers shown

Title
ABBA: Highly Expressive Hadamard Product Adaptation for Large Language Models Raghav Singhal Kaustubh Ponkshe Rohit Vartak Praneeth Vepakomma 7 0 0 20 May 2025
SRLoRA: Subspace Recomposition in Low-Rank Adaptation via Importance-Based Fusion and Reinitialization Haodong Yang Lei Wang Md Zakir Hossain 12 0 0 18 May 2025
AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projections Xin Yu Yujia Wang Jinghui Chen Lingzhou Xue 17 0 0 18 May 2025
Memory-Efficient LLM Training by Various-Grained Low-Rank Projection of Gradients Yezhen Wang Zhouhao Yang Brian K Chen Fanyi Pu Bo-wen Li Tianyu Gao Kenji Kawaguchi 46 0 0 03 May 2025
Generalized Tensor-based Parameter-Efficient Fine-Tuning via Lie Group Transformations Chongjie Si Zhiyi Shi Xuehui Wang Yichen Xiao Xiaokang Yang Wei-Ming Shen AI4CE 70 0 0 01 Apr 2025
Hiding Images in Diffusion Models by Editing Learned Score Functions Haoyu Chen Yunqiao Yang Nan Zhong Kede Ma DiffM 68 0 0 24 Mar 2025
Iterative Prompt Relocation for Distribution-Adaptive Visual Prompt Tuning Chikai Shang Mengke Li Yiqun Zhang Zhen Chen Jinlin Wu Fangqing Gu Yang Lu Yiu-ming Cheung VLM 74 0 0 10 Mar 2025
R-LoRA: Random Initialization of Multi-Head LoRA for Multi-Task Learning Jinda Liu Yi-Ju Chang Yuan Wu 59 0 0 24 Feb 2025
Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment Chenghao Fan Zhenyi Lu Sichen Liu Xiaoye Qu Wei Wei Chengfeng Gu Yu-Xi Cheng MoE 183 0 0 24 Feb 2025
A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models Mengyang Sun Yihao Wang Tao Feng Dan Zhang Yifan Zhu J. Tang MoE 43 0 0 20 Feb 2025
GoRA: Gradient-driven Adaptive Low Rank Adaptation Haonan He Peng Ye Yuchen Ren Yuan Yuan Lei Chen AI4TS AI4CE 211 0 0 13 Feb 2025
LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits Zikai Zhou Qizheng Zhang Hermann Kumbong Kunle Olukotun MQ 288 0 0 12 Feb 2025
DAGPrompT: Pushing the Limits of Graph Prompting with a Distribution-aware Graph Prompt Tuning Approach Qin Chen Liang Wang Bo Zheng Guojie Song 41 2 0 28 Jan 2025
A Hessian-informed hyperparameter optimization for differential learning rate Shiyun Xu Zhiqi Bu Yiliang Zhang Ian Barnett 39 1 0 12 Jan 2025
DoTA: Weight-Decomposed Tensor Adaptation for Large Language Models Xiaolin Hu Xiang Cheng Peiyu Liu Wei Liu Jian Luan Bin Wang Y. Liu 44 1 0 31 Dec 2024
Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuning Kaustubh Ponkshe Raghav Singhal Eduard A. Gorbunov Alexey Tumanov Samuel Horváth Praneeth Vepakomma 74 2 0 29 Nov 2024
COAP: Memory-Efficient Training with Correlation-Aware Gradient Projection Jinqi Xiao S. Sang Tiancheng Zhi Jing Liu Qing Yan Linjie Luo Bo Yuan Bo Yuan VLM 103 1 0 26 Nov 2024
On the Crucial Role of Initialization for Matrix Factorization Bingcong Li Liang Zhang Aryan Mokhtari Niao He 31 1 0 24 Oct 2024
Parameter Efficient Fine-tuning via Explained Variance Adaptation Fabian Paischer Lukas Hauzenberger Thomas Schmied Benedikt Alkin Marc Peter Deisenroth Sepp Hochreiter 37 4 0 09 Oct 2024
Flat-LoRA: Low-Rank Adaption over a Flat Loss Landscape Tao Li Zhengbao He Yujun Li Yasheng Wang Lifeng Shang X. Huang 53 0 0 22 Sep 2024
LoRA-Pro: Are Low-Rank Adapters Properly Optimized? Zhengbo Wang Jian Liang Ran He Zilei Wang Tieniu Tan 73 16 0 25 Jul 2024
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey Zeyu Han Chao Gao Jinyang Liu Jeff Zhang Sai Qian Zhang 150 318 0 21 Mar 2024
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 366 12,003 0 04 Mar 2022
The Power of Scale for Parameter-Efficient Prompt Tuning Brian Lester Rami Al-Rfou Noah Constant VPVLM 280 3,858 0 18 Apr 2021
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 299 6,984 0 20 Apr 2018