Higher Layers Need More LoRA Experts

13 February 2024

Ruibo Liu

Jie Yang

Papers citing "Higher Layers Need More LoRA Experts"

36 / 36 papers shown

Title
A Sensitivity-Driven Expert Allocation Method in LoRA-MoE for Efficient Fine-Tuning Junzhou Xu Boyu Diao MoE 52 0 0 06 May 2025
TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Experts Pradip Kunwar Minh Vu Maanak Gupta Mahmoud Abdelsalam Manish Bhattarai MoE MoMe 187 0 0 29 Apr 2025
DynMoLE: Boosting Mixture of LoRA Experts Fine-Tuning with a Hybrid Routing Mechanism Dengchun Li Naizheng Wang Zihao Zhang Haoyang Yin Lei Duan Meng Xiao Mingjie Tang MoE 56 1 0 01 Apr 2025
Mixture of Routers Jia-Chen Zhang Yu-Jie Xiong Xi-He Qiu Chun-Ming Xia Fei Dai MoE 76 0 0 30 Mar 2025
Not All LoRA Parameters Are Essential: Insights on Inference Necessity Guanhua Chen Yutong Yao Ci-Jun Gao Lidia S. Chao Feng Wan Derek F. Wong 39 0 0 30 Mar 2025
A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models Mengyang Sun Yihao Wang Tao Feng Dan Zhang Yifan Zhu J. Tang MoE 43 0 0 20 Feb 2025
Rank Also Matters: Hierarchical Configuration for Mixture of Adapter Experts in LLM Fine-Tuning Peizhuang Cong Wenpu Liu Wenhan Yu Haochen Zhao Tong Yang ALM MoE 81 0 0 06 Feb 2025
Ensembles of Low-Rank Expert Adapters Yinghao Li Vianne Gao Chao Zhang MohamadAli Torkamani 75 0 0 31 Jan 2025
GraphLoRA: Empowering LLMs Fine-Tuning via Graph Collaboration of MoE Ting Bai Yue Yu Le Huang Zenan Xu Zhe Zhao Chuan Shi MoE 230 0 0 18 Dec 2024
MoSLD: An Extremely Parameter-Efficient Mixture-of-Shared LoRAs for Multi-Task Learning Lulu Zhao Weihao Zeng Xiaofeng Shi Hua Zhou MoMe MoE 83 0 0 12 Dec 2024
Personalized Federated Fine-Tuning for LLMs via Data-Driven Heterogeneous Model Architectures Yicheng Zhang Zhen Qin Zhaomin Wu Jian Hou Shuiguang Deng 80 2 0 28 Nov 2024
MALoRA: Mixture of Asymmetric Low-Rank Adaptation for Enhanced Multi-Task Learning Xujia Wang Haiyan Zhao Shuo Wang Hanqing Wang Zhiyuan Liu MoMe MoE 40 0 0 30 Oct 2024
AdaptGCD: Multi-Expert Adapter Tuning for Generalized Category Discovery Yuxun Qu Yongqiang Tang Chenyang Zhang Wensheng Zhang 31 0 0 29 Oct 2024
Towards Optimal Adapter Placement for Efficient Transfer Learning Aleksandra I. Nowak Otniel-Bogdan Mercea Anurag Arnab Jonas Pfeiffer Yann N. Dauphin Utku Evci 40 0 0 21 Oct 2024
AlphaLoRA: Assigning LoRA Experts Based on Layer Training Quality Peijun Qing Chongyang Gao Yefan Zhou Xingjian Diao Yaoqing Yang Soroush Vosoughi MoMe MoE 24 4 0 14 Oct 2024
SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture Jiayi Han Liang Du Hongwei Du Xiangguo Zhou Yiwen Wu Weibo Zheng Donghong Han CLL MoMe MoE 38 2 0 10 Oct 2024
DLP-LoRA: Efficient Task-Specific LoRA Fusion with a Dynamic, Lightweight Plugin for Large Language Models Yuxuan Zhang Ruizhe Li MoMe 58 0 0 02 Oct 2024
LoraMap: Harnessing the Power of LoRA Connections Hyeryun Park Jeongwon Kwak Dongsuk Jang Sumin Park Jinwook Choi MoMe 33 0 0 29 Aug 2024
Low-Rank Interconnected Adaptation Across Layers Yibo Zhong Yao Zhou OffRL MoE 48 1 0 13 Jul 2024
A Survey on LoRA of Large Language Models Yuren Mao Yuhang Ge Yijiang Fan Wenyi Xu Yu Mi Zhonghao Hu Yunjun Gao ALM 58 25 0 08 Jul 2024
Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learning Ziyu Zhao Leilei Gan Guoyin Wang Yuwei Hu Tao Shen Hongxia Yang Kun Kuang Fei Wu MoE MoMe 39 12 0 24 Jun 2024
AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Models Zihao Zeng Yibo Miao Hongcheng Gao Hao Zhang Zhijie Deng MoE 52 8 0 19 Jun 2024
MEMoE: Enhancing Model Editing with Mixture of Experts Adaptors Renzhi Wang Piji Li KELM 42 3 0 29 May 2024
MLAE: Masked LoRA Experts for Parameter-Efficient Fine-Tuning Junjie Wang Guangjing Yang Wentao Chen Huahui Yi Xiaohu Wu Qicheng Lao MoE ALM 44 0 0 29 May 2024
LoRA-Switch: Boosting the Efficiency of Dynamic LLM Adapters via System-Algorithm Co-design Rui Kong Qiyang Li Xinyu Fang Qingtian Feng Qingfeng He Yazhu Dong Weijun Wang Yuanchun Li Linghe Kong Yunxin Liu MoE 40 4 0 28 May 2024
MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priors Yuan Tang Xu Han Xianzhi Li Qiao Yu Yixue Hao Long Hu Min Chen 37 14 0 02 May 2024
AdaMoLE: Fine-Tuning Large Language Models with Adaptive Mixture of Low-Rank Adaptation Experts Zefang Liu Jiahua Luo MoE KELM 43 11 0 01 May 2024
MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts Dengchun Li Yingzi Ma Naizheng Wang Zhengmao Ye Zhiyuan Cheng ... Yan Zhang Lei Duan Jie Zuo Cal Yang Mingjie Tang MoE 40 44 0 22 Apr 2024
Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts Weilin Cai Juyong Jiang Le Qin Junwei Cui Sunghun Kim Jiayi Huang 57 7 0 07 Apr 2024
Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learning Huiyi Wang Haodong Lu Lina Yao Dong Gong KELM CLL 48 9 0 27 Mar 2024
MoSA: Mixture of Sparse Adapters for Visual Efficient Tuning Qizhe Zhang Bocheng Zou Ruichuan An Jiaming Liu Shanghang Zhang MoE 29 2 0 05 Dec 2023
JudgeLM: Fine-tuned Large Language Models are Scalable Judges Lianghui Zhu Xinggang Wang Xinlong Wang ELM ALM 62 112 0 26 Oct 2023
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering Pan Lu Swaroop Mishra Tony Xia Liang Qiu Kai-Wei Chang Song-Chun Zhu Oyvind Tafjord Peter Clark Ashwin Kalyan ELM ReLM LRM 211 1,124 0 20 Sep 2022
Multitask Prompted Training Enables Zero-Shot Task Generalization Victor Sanh Albert Webson Colin Raffel Stephen H. Bach Lintang Sutawika ... T. Bers Stella Biderman Leo Gao Thomas Wolf Alexander M. Rush LRM 215 1,663 0 15 Oct 2021
The Power of Scale for Parameter-Efficient Prompt Tuning Brian Lester Rami Al-Rfou Noah Constant VPVLM 280 3,872 0 18 Apr 2021
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 299 6,984 0 20 Apr 2018