RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptation

RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptation

9 January 2024

Elvir Crnčević

Dan Alistarh

Papers citing "RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptation"

18 / 18 papers shown

Title
An Efficient Row-Based Sparse Fine-Tuning Cen-Jhih Li Aditya Bhaskara 56 0 0 17 Feb 2025
Sparse High Rank Adapters K. Bhardwaj N. Pandey Sweta Priyadarshi Viswanath Ganapathy Rafael Esteves ... P. Whatmough Risheek Garrepalli M. V. Baalen Harris Teague Markus Nagel MQ 43 4 0 28 Jan 2025
SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs Mohammad Mozaffari Amir Yazdanbakhsh Zhao Zhang M. Dehnavi 78 5 0 28 Jan 2025
Spectral-Aware Low-Rank Adaptation for Speaker Verification Zhe Li Man-Wai Mak Mert Pilanci Hung-yi Lee Helen Meng 41 0 0 07 Jan 2025
GaLore $+$ : Boosting Low-Rank Adaptation for LLMs with Cross-Head Projection Xutao Liao Shaohui Li Yuhui Xu Zhi Li Y. Liu You He VLM 59 2 0 31 Dec 2024
RECAST: Reparameterized, Compact weight Adaptation for Sequential Tasks Nazia Tasnim Bryan A. Plummer CLL OffRL 74 0 0 25 Nov 2024
LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics Thomas Robert M. Safaryan Ionut-Vlad Modoranu Dan Alistarh ODL 36 2 0 21 Oct 2024
Mitigating Forgetting in LLM Supervised Fine-Tuning and Preference Learning H. Fernando Han Shen Parikshit Ram Yi Zhou Horst Samulowitz Nathalie Baracaldo Tianyi Chen CLL 56 2 0 20 Oct 2024
Two Sparse Matrices are Better than One: Sparsifying Neural Networks with Double Sparse Factorization Vladimír Boža Vladimír Macko 25 1 0 27 Sep 2024
A Bayesian Interpretation of Adaptive Low-Rank Adaptation Haolin Chen Philip N. Garner 55 1 0 16 Sep 2024
Efficient Quantized Sparse Matrix Operations on Tensor Cores Shigang Li Kazuki Osawa Torsten Hoefler 82 31 0 14 Sep 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 328 11,953 0 04 Mar 2022
Multitask Prompted Training Enables Zero-Shot Task Generalization Victor Sanh Albert Webson Colin Raffel Stephen H. Bach Lintang Sutawika ... T. Bers Stella Biderman Leo Gao Thomas Wolf Alexander M. Rush LRM 213 1,657 0 15 Oct 2021
P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks Xiao Liu Kaixuan Ji Yicheng Fu Weng Lam Tam Zhengxiao Du Zhilin Yang Jie Tang VLM 238 806 0 14 Oct 2021
RAFT: A Real-World Few-Shot Text Classification Benchmark Neel Alex Eli Lifland Lewis Tunstall A. Thakur Pegah Maham ... Carolyn Ashurst Paul Sedille A. Carlier M. Noetel Andreas Stuhlmuller RALM 184 56 0 28 Sep 2021
The Power of Scale for Parameter-Efficient Prompt Tuning Brian Lester Rami Al-Rfou Noah Constant VPVLM 280 3,848 0 18 Apr 2021
Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks Itay Hubara Brian Chmiel Moshe Island Ron Banner S. Naor Daniel Soudry 53 111 0 16 Feb 2021
Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks Torsten Hoefler Dan Alistarh Tal Ben-Nun Nikoli Dryden Alexandra Peste MQ 141 684 0 31 Jan 2021