AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning

31 October 2022

Yaqing Wang

Sahaj Agarwal

Subhabrata Mukherjee

Xiaodong Liu

Jing Gao

Ahmed Hassan Awadallah

Jianfeng Gao

MoE

ArXiv PDF HTML

Papers citing "AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning"

36 / 86 papers shown

Title
BECoTTA: Input-dependent Online Blending of Experts for Continual Test-time Adaptation Daeun Lee Jaehong Yoon Sung Ju Hwang CLL TTA 62 5 0 13 Feb 2024
Learning to Route Among Specialized Experts for Zero-Shot Generalization Mohammed Muqeeth Haokun Liu Yufan Liu Colin Raffel MoMe 37 34 0 08 Feb 2024
Efficient Fine-tuning of Audio Spectrogram Transformers via Soft Mixture of Adapters Umberto Cappellazzo Daniele Falavigna A. Brutti MoE 38 2 0 01 Feb 2024
Institutional Platform for Secure Self-Service Large Language Model Exploration V. Bumgardner Mitchell A. Klusty W. V. Logan Samuel E. Armstrong Caylin D. Hickey Jeff Talbert Caylin Hickey Jeff Talbert 56 1 0 01 Feb 2024
X-PEFT: eXtremely Parameter-Efficient Fine-Tuning for Extreme Multi-Profile Scenarios Namju Kwak Taesup Kim MoE 21 0 0 29 Jan 2024
What the Weight?! A Unified Framework for Zero-Shot Knowledge Composition Carolin Holtermann Markus Frohmann Navid Rekabsaz Anne Lauscher MoMe 24 5 0 23 Jan 2024
PersianMind: A Cross-Lingual Persian-English Large Language Model Pedram Rostami Ali Salemi M. Dousti CLL LRM 24 5 0 12 Jan 2024
Empirical Analysis of Efficient Fine-Tuning Methods for Large Pre-Trained Language Models Nigel Doering Cyril Gorlla Trevor Tuttle Adhvaith Vijay 22 1 0 08 Jan 2024
Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuning Yunhao Gou Zhili Liu Kai Chen Lanqing Hong Hang Xu Aoxue Li Dit-Yan Yeung James T. Kwok Yu Zhang MoE MLLM VLM 36 62 0 19 Dec 2023
Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment Lingling Xu Haoran Xie S. J. Qin Xiaohui Tao F. Wang 49 135 0 19 Dec 2023
MoSA: Mixture of Sparse Adapters for Visual Efficient Tuning Qizhe Zhang Bocheng Zou Ruichuan An Jiaming Liu Shanghang Zhang MoE 27 2 0 05 Dec 2023
Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts Jialin Wu Xia Hu Yaqing Wang Bo Pang Radu Soricut MoE 19 14 0 01 Dec 2023
Towards Better Parameter-Efficient Fine-Tuning for Large Language Models: A Position Paper Chengyu Wang Junbing Yan Wei Zhang Jun Huang ALM 42 3 0 22 Nov 2023
MultiLoRA: Democratizing LoRA for Better Multi-Task Learning Yiming Wang Yu Lin Xiaodong Zeng Guannan Zhang MoMe 44 20 0 20 Nov 2023
Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learning Clifton A. Poth Hannah Sterz Indraneil Paul Sukannya Purkayastha Leon Arne Engländer Timo Imhof Ivan Vulić Sebastian Ruder Iryna Gurevych Jonas Pfeiffer 32 45 0 18 Nov 2023
SiRA: Sparse Mixture of Low Rank Adaptation Yun Zhu Nevan Wichers Chu-Cheng Lin Xinyi Wang Tianlong Chen ... Han Lu Canoee Liu Liangchen Luo Jindong Chen Lei Meng MoE 25 27 0 15 Nov 2023
Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE Zeren Chen Ziqin Wang Zhen Wang Huayang Liu Zhen-fei Yin Si Liu Lu Sheng Wanli Ouyang Yu Qiao Jing Shao MoE 36 7 0 05 Nov 2023
One For All & All For One: Bypassing Hyperparameter Tuning with Model Averaging For Cross-Lingual Transfer Fabian David Schmidt Ivan Vulić Goran Glavavs MoMe 16 3 0 16 Oct 2023
Decomposed Prompt Tuning via Low-Rank Reparameterization Yao Xiao Lu Xu Jiaxi Li Wei Lu Xiaoli Li VLM 22 6 0 16 Oct 2023
OWL: A Large Language Model for IT Operations Hongcheng Guo Jian Yang Jiaheng Liu Liqun Yang Linzheng Chai ... Tieqiao Zheng Liangfan Zheng Bo-Wen Zhang Ke Xu Zhoujun Li VLM 66 41 0 17 Sep 2023
Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning Ted Zadouri A. Ustun Arash Ahmadian Beyza Ermics Acyr F. Locatelli Sara Hooker MoE 35 88 0 11 Sep 2023
SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models Sara Babakniya A. Elkordy Yahya H. Ezzeldin Qingfeng Liu Kee-Bong Song Mostafa El-Khamy Salman Avestimehr 21 56 0 12 Aug 2023
LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition Chengsong Huang Qian Liu Bill Yuchen Lin Tianyu Pang Chao Du Min-Bin Lin MoMe 38 185 0 25 Jul 2023
PRISMA-DFLLM: An Extension of PRISMA for Systematic Literature Reviews using Domain-specific Finetuned Large Language Models Teo Susnjak 19 11 0 15 Jun 2023
Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models Memories Shizhe Diao Tianyang Xu Ruijia Xu Jiawei Wang Tong Zhang MoE AI4CE 13 36 0 08 Jun 2023
Free Lunch: Robust Cross-Lingual Transfer via Model Checkpoint Averaging Fabian David Schmidt Ivan Vulić Goran Glavavs 24 8 0 26 May 2023
DADA: Dialect Adaptation via Dynamic Aggregation of Linguistic Rules Yanchen Liu William B. Held Diyi Yang 53 10 0 22 May 2023
Parameter-Efficient Fine-Tuning with Layer Pruning on Free-Text Sequence-to-Sequence Modeling Y. Zhu Xuebing Yang Yuanyuan Wu Wensheng Zhang MedIm 20 2 0 15 May 2023
Text-guided High-definition Consistency Texture Model Zhibin Tang Tiantong He DiffM 23 6 0 10 May 2023
Modular Deep Learning Jonas Pfeiffer Sebastian Ruder Ivan Vulić E. Ponti MoMe OOD 32 73 0 22 Feb 2023
AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuning Han Zhou Xingchen Wan Ivan Vulić Anna Korhonen 21 45 0 28 Jan 2023
Efficient Methods for Natural Language Processing: A Survey Marcos Vinícius Treviso Ji-Ung Lee Tianchu Ji Betty van Aken Qingqing Cao ... Emma Strubell Niranjan Balasubramanian Leon Derczynski Iryna Gurevych Roy Schwartz 28 109 0 31 Aug 2022
The Power of Scale for Parameter-Efficient Prompt Tuning Brian Lester Rami Al-Rfou Noah Constant VPVLM 280 3,848 0 18 Apr 2021
Making Pre-trained Language Models Better Few-shot Learners Tianyu Gao Adam Fisch Danqi Chen 241 1,919 0 31 Dec 2020
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 297 6,959 0 20 Apr 2018
Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning Y. Gal Zoubin Ghahramani UQCV BDL 285 9,138 0 06 Jun 2015