LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters

LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters

27 May 2024

Klaudia Bałazy

Mohammadreza Banaei

Papers citing "LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters"

14 / 14 papers shown

Title
ABBA: Highly Expressive Hadamard Product Adaptation for Large Language Models Raghav Singhal Kaustubh Ponkshe Rohit Vartak Praneeth Vepakomma 59 0 0 20 May 2025
Model Diffusion for Certifiable Few-shot Transfer Learning Fady Rezk Royson Lee Henry Gouk Timothy M. Hospedales Minyoung Kim 86 0 0 10 Feb 2025
Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuning Kaustubh Ponkshe Raghav Singhal Eduard A. Gorbunov Alexey Tumanov Samuel Horváth Praneeth Vepakomma 160 6 0 29 Nov 2024
Talking Heads: Understanding Inter-layer Communication in Transformer Language Models Jack Merullo Carsten Eickhoff Ellie Pavlick 84 14 0 13 Jun 2024
PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models Fanxu Meng Zhaohui Wang Muhan Zhang VLM 98 92 0 03 Apr 2024
LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models Zhiqiang Hu Lei Wang Yihuai Lan Wanyu Xu Ee-Peng Lim Lidong Bing Xing Xu Soujanya Poria Roy Ka-wei Lee ALM 73 249 0 04 Apr 2023
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models Elad Ben-Zaken Shauli Ravfogel Yoav Goldberg 135 1,191 0 18 Jun 2021
LoRA: Low-Rank Adaptation of Large Language Models J. E. Hu Yelong Shen Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang Weizhu Chen OffRL AI4TS AI4CE ALM AIMat 223 9,946 0 17 Jun 2021
The Power of Scale for Parameter-Efficient Prompt Tuning Brian Lester Rami Al-Rfou Noah Constant VPVLM 448 3,952 0 18 Apr 2021
Prefix-Tuning: Optimizing Continuous Prompts for Generation Xiang Lisa Li Percy Liang 164 4,167 0 01 Jan 2021
AdapterFusion: Non-Destructive Task Composition for Transfer Learning Jonas Pfeiffer Aishwarya Kamath Andreas Rucklé Kyunghyun Cho Iryna Gurevych CLL MoMe 113 837 0 01 May 2020
PyTorch: An Imperative Style, High-Performance Deep Learning Library Adam Paszke Sam Gross Francisco Massa Adam Lerer James Bradbury ... Sasank Chilamkurthy Benoit Steiner Lu Fang Junjie Bai Soumith Chintala ODL 231 42,038 0 03 Dec 2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy M. Lewis Luke Zettlemoyer Veselin Stoyanov AIMat 392 24,160 0 26 Jul 2019
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions Christopher Clark Kenton Lee Ming-Wei Chang Tom Kwiatkowski Michael Collins Kristina Toutanova 169 1,475 0 24 May 2019