A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA

28 November 2023

Papers citing "A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA"

13 / 63 papers shown

Title
Rapid Switching and Multi-Adapter Fusion via Sparse High Rank Adapters Kartikeya Bhardwaj N. Pandey Sweta Priyadarshi Viswanath Ganapathy Rafael Esteves ... Paul N. Whatmough Risheek Garrepalli M. V. Baalen Harris Teague Markus Nagel MoMe 33 0 0 22 Jul 2024
LoRA-GA: Low-Rank Adaptation with Gradient Approximation Shaowen Wang Linxi Yu Jian Li ALM AI4CE 26 29 0 06 Jul 2024
Structured Unrestricted-Rank Matrices for Parameter Efficient Fine-tuning Arijit Sehanobish Avinava Dubey Krzysztof Choromanski Somnath Basu Roy Chowdhury Deepali Jain Vikas Sindhwani Snigdha Chaturvedi ALM 43 1 0 25 Jun 2024
Fighting Randomness with Randomness: Mitigating Optimisation Instability of Fine-Tuning using Delayed Ensemble and Noisy Interpolation Branislav Pecher Ján Cegin Róbert Belanec Jakub Simko Ivan Srba M. Bieliková 44 1 0 18 Jun 2024
FouRA: Fourier Low Rank Adaptation Shubhankar Borse Shreya Kadambi N. Pandey Kartikeya Bhardwaj Viswanath Ganapathy Sweta Priyadarshi Risheek Garrepalli Rafael Esteves Munawar Hayat Fatih Porikli 42 6 0 13 Jun 2024
MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning Hanqing Wang Zeguan Xiao Shuo Wang Guanhua Chen Guanhua Chen 44 19 0 13 Jun 2024
The Impact of Initialization on LoRA Finetuning Dynamics Soufiane Hayou Nikhil Ghosh Bin Yu AI4CE 36 11 0 12 Jun 2024
OLoRA: Orthonormal Low-Rank Adaptation of Large Language Models Kerim Büyükakyüz AI4CE 21 5 0 03 Jun 2024
LoRA Learns Less and Forgets Less D. Biderman Jose Javier Gonzalez Ortiz Jacob P. Portes Mansheej Paul Philip Greengard ... Sam Havens Vitaliy Chiley Jonathan Frankle Cody Blakeney John P. Cunningham CLL 35 110 0 15 May 2024
In-Context Learning with Long-Context Models: An In-Depth Exploration Amanda Bertsch Maor Ivgi Uri Alon Jonathan Berant Matthew R. Gormley Matthew R. Gormley Graham Neubig ReLM AIMat 93 64 0 30 Apr 2024
LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models Yaowei Zheng Richong Zhang Junhao Zhang Yanhan Ye Zheyan Luo Zhangchi Feng Yongqiang Ma 46 383 0 20 Mar 2024
Scaling Laws for Forgetting When Fine-Tuning Large Language Models Damjan Kalajdzievski CLL 36 8 0 11 Jan 2024
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 319 11,953 0 04 Mar 2022