Sparse Low-rank Adaptation of Pre-trained Language Models

Sparse Low-rank Adaptation of Pre-trained Language Models

20 November 2023

Zhiyuan Liu

Maosong Sun

Papers citing "Sparse Low-rank Adaptation of Pre-trained Language Models"

17 / 17 papers shown

Title
Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device Learning Le-Trung Nguyen Ael Quélennec Van-Tam Nguyen Enzo Tartaglione 48 0 0 08 May 2025
Communication-Efficient and Personalized Federated Foundation Model Fine-Tuning via Tri-Matrix Adaptation Yongqian Li Bo Liu Sheng Huang Zhe Zhang Xiaotong Yuan Richang Hong 51 0 0 31 Mar 2025
DeLoRA: Decoupling Angles and Strength in Low-rank Adaptation Massimo Bini Leander Girrbach Zeynep Akata 45 0 0 23 Mar 2025
Parameter-Efficient Fine-Tuning of Large Language Models via Deconvolution in Subspace Jia-Chen Zhang Yu-Jie Xiong Chun-Ming Xia Dong-Hai Zhu Xi-He Qiu 66 1 0 03 Mar 2025
Dynamic Low-Rank Sparse Adaptation for Large Language Models Weizhong Huang Yuxin Zhang Xiawu Zheng Yu Liu Jing Lin Yiwu Yao Rongrong Ji 97 1 0 21 Feb 2025
Sparse High Rank Adapters K. Bhardwaj N. Pandey Sweta Priyadarshi Viswanath Ganapathy Rafael Esteves ... P. Whatmough Risheek Garrepalli M. V. Baalen Harris Teague Markus Nagel MQ 43 4 0 28 Jan 2025
Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learning Ziyu Zhao Yixiao Zhou Didi Zhu Tao Shen Xinze Wang Jing Su Kun Kuang Zhongyu Wei Fei Wu Yu Cheng MoE 45 2 0 28 Jan 2025
BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models Yibin Wang Haochen Shi Ligong Han Dimitris N. Metaxas Hao Wang BDL UQLM 116 8 0 28 Jan 2025
Adaptive Rank, Reduced Forgetting: Knowledge Retention in Continual Learning Vision-Language Models with Dynamic Rank-Selective LoRA Haodong Lu Chongyang Zhao Jason Xue Lina Yao Kristen Moore Dong Gong VLM KELM CLL 92 4 0 01 Dec 2024
Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologies Liwen Wang Sheng Chen Linnan Jiang Shu Pan Runze Cai Sen Yang Fei Yang 52 3 0 24 Oct 2024
Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely Siyun Zhao Yuqing Yang Zilong Wang Zhiyuan He Luna Qiu Lili Qiu SyDa RALM 3DV 46 36 0 23 Sep 2024
A Bayesian Interpretation of Adaptive Low-Rank Adaptation Haolin Chen Philip N. Garner 58 1 0 16 Sep 2024
Federated LoRA with Sparse Communication Kevin Kuo Arian Raje Kousik Rajesh Virginia Smith 46 8 0 07 Jun 2024
Learning Discrete Concepts in Latent Hierarchical Models Lingjing Kong Guan-Hong Chen Erdun Gao Eric P. Xing Yuejie Chi Kun Zhang 52 4 0 01 Jun 2024
Single-dataset Experts for Multi-dataset Question Answering Dan Friedman Ben Dodge Danqi Chen MoMe 134 26 0 28 Sep 2021
The Power of Scale for Parameter-Efficient Prompt Tuning Brian Lester Rami Al-Rfou Noah Constant VPVLM 280 3,872 0 18 Apr 2021
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 299 6,996 0 20 Apr 2018