AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Models

14 February 2023

Alexandra Chronopoulou

Papers citing "AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Models"

50 / 55 papers shown

Title
FedMerge: Federated Personalization via Model Merging Shutong Chen Tianyi Zhou Guodong Long Jing Jiang Chengqi Zhang FedML MoMe 49 0 0 09 Apr 2025
MASS: MoErging through Adaptive Subspace Selection Donato Crisostomi Alessandro Zirilli Antonio Andrea Gargiulo Maria Sofia Bucarelli Simone Scardapane Fabrizio Silvestri Iacopo Masi Emanuele Rodolà MoMe 40 0 0 06 Apr 2025
Semantic Library Adaptation: LoRA Retrieval and Fusion for Open-Vocabulary Semantic Segmentation Reza Qorbani Gianluca Villani Theodoros Panagiotakopoulos Marc Botet Colomer Linus Harenstam-Nielsen ... Pier Luigi Dovesi Jussi Karlgren Daniel Cremers F. Tombari Matteo Poggi VLM 52 0 0 27 Mar 2025
Serial Low-rank Adaptation of Vision Transformer Houqiang Zhong Shaocheng Shen Ke Cai Zhenglong Wu Jiangchao Yao Yuan Cheng Xuefei Li Xiaoyun Zhang Li-Na Song Qiang Hu 47 0 0 22 Mar 2025
NEAT: Nonlinear Parameter-efficient Adaptation of Pre-trained Models Yibo Zhong Haoxiang Jiang Lincan Li Ryumei Nakada Tianci Liu Linjun Zhang Huaxiu Yao Haoyu Wang 77 2 0 24 Feb 2025
Generative Modeling of Individual Behavior at Scale Nabil Omi Lucas Caccia Anurag Sarkar Jordan T. Ash S. Sen 76 0 0 24 Feb 2025
Propagation of Chaos for Mean-Field Langevin Dynamics and its Application to Model Ensemble Atsushi Nitanda Anzelle Lee Damian Tan Xing Kai Mizuki Sakaguchi Taiji Suzuki AI4CE 64 1 0 09 Feb 2025
Ensembles of Low-Rank Expert Adapters Yinghao Li Vianne Gao Chao Zhang MohamadAli Torkamani 69 0 0 31 Jan 2025
Adapter-based Approaches to Knowledge-enhanced Language Models -- A Survey Alexander Fichtl Juraj Vladika Georg Groh KELM 80 0 0 25 Nov 2024
A Practical Guide to Fine-tuning Language Models with Limited Data Márton Szép Daniel Rueckert Rüdiger von Eisenhart-Rothe Florian Hinterwimmer SyDa ALM 49 2 0 14 Nov 2024
Model Fusion through Bayesian Optimization in Language Model Fine-Tuning Chaeyun Jang Hyungi Lee Jungtaek Kim Juho Lee MoMe 45 0 0 11 Nov 2024
Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologies L. Wang Sheng Chen Linnan Jiang Shu Pan Runze Cai Sen Yang Fei Yang 49 3 0 24 Oct 2024
Mitigating the Backdoor Effect for Multi-Task Model Merging via Safety-Aware Subspace Jinluan Yang Anke Tang Didi Zhu Zhengyu Chen Li Shen Fei Wu MoMe AAML 62 3 0 17 Oct 2024
CollabEdit: Towards Non-destructive Collaborative Knowledge Editing Jiamu Zheng Jinghuai Zhang Tianyu Du Xuhong Zhang Jianwei Yin Tao Lin KELM 40 0 0 12 Oct 2024
Glider: Global and Local Instruction-Driven Expert Router Pingzhi Li Prateek Yadav Jaehong Yoon Jie Peng Yi-Lin Sung Joey Tianyi Zhou Tianlong Chen MoMe MoE 33 1 0 09 Oct 2024
DaWin: Training-free Dynamic Weight Interpolation for Robust Adaptation Changdae Oh Yixuan Li Kyungwoo Song Sangdoo Yun Dongyoon Han OOD MoMe 45 4 0 03 Oct 2024
Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering Ziyu Zhao Tao Shen Didi Zhu Zexi Li Jing Su Xuwu Wang Kun Kuang Fei Wu MoMe 36 6 0 24 Sep 2024
Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely Siyun Zhao Yuqing Yang Zilong Wang Zhiyuan He Luna Qiu Lili Qiu SyDa RALM 3DV 44 35 0 23 Sep 2024
LLM-based multi-agent poetry generation in non-cooperative environments Ran Zhang Steffen Eger LLMAG 37 5 0 05 Sep 2024
Leveraging Open Knowledge for Advancing Task Expertise in Large Language Models Yuncheng Yang Yulei Qin Tong Wu Zihan Xu Gang Li ... Yuchen Shi Ke Li Xing Sun Jie Yang Yun Gu ALM OffRL MoE 52 0 0 28 Aug 2024
MergeRepair: An Exploratory Study on Merging Task-Specific Adapters in Code LLMs for Automated Program Repair Meghdad Dehghan Jie JW Wu Fatemeh H. Fard Ali Ouni MoMe 50 2 0 18 Aug 2024
A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learning Prateek Yadav Colin Raffel Mohammed Muqeeth Lucas Caccia Haokun Liu Tianlong Chen Joey Tianyi Zhou Leshem Choshen Alessandro Sordoni MoMe 46 21 0 13 Aug 2024
Learning Scalable Model Soup on a Single GPU: An Efficient Subspace Training Strategy Tao Li Weisen Jiang Fanghui Liu X. Huang James T. Kwok MoMe 63 1 0 04 Jul 2024
Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learning Ziyu Zhao Leilei Gan Guoyin Wang Yuwei Hu Tao Shen Hongxia Yang Kun Kuang Fei Wu MoE MoMe 39 11 0 24 Jun 2024
Towards Efficient Pareto Set Approximation via Mixture of Experts Based Model Fusion Anke Tang Li Shen Yong Luo Shiwei Liu Han Hu Bo Du MoMe 31 6 0 14 Jun 2024
FusionBench: A Comprehensive Benchmark of Deep Model Fusion Anke Tang Li Shen Yong Luo Han Hu Bo Du Dacheng Tao ELM MoMe VLM 44 22 0 05 Jun 2024
Low-Rank Adaption on Transformer-based Oriented Object Detector for Satellite Onboard Processing of Remote Sensing Images Xinyang Pu Feng Xu 32 3 0 04 Jun 2024
Spectral Adapter: Fine-Tuning in Spectral Space Fangzhao Zhang Mert Pilanci 42 9 0 22 May 2024
Towards Modular LLMs by Building and Reusing a Library of LoRAs O. Ostapenko Zhan Su E. Ponti Laurent Charlin Nicolas Le Roux Matheus Pereira Lucas Caccia Alessandro Sordoni MoMe 41 31 0 18 May 2024
Stylus: Automatic Adapter Selection for Diffusion Models Michael Luo Justin Wong Brandon Trabucco Yanping Huang Joseph E. Gonzalez Zhifeng Chen Ruslan Salakhutdinov Ion Stoica DiffM 43 6 0 29 Apr 2024
No Train but Gain: Language Arithmetic for training-free Language Adapters enhancement Mateusz Klimaszewski Piotr Andruszkiewicz Alexandra Birch MoMe 47 4 0 24 Apr 2024
In-Context Learning State Vector with Inner and Momentum Optimization Dongfang Li Zhenyu Liu Xinshuo Hu Zetian Sun Baotian Hu Min Zhang 37 5 0 17 Apr 2024
AdapterSwap: Continuous Training of LLMs with Data Removal and Access-Control Guarantees William Fleshman Aleem Khan Marc Marone Benjamin Van Durme CLL KELM 58 3 0 12 Apr 2024
PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models Fanxu Meng Zhaohui Wang Muhan Zhang VLM 64 73 0 03 Apr 2024
Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learning Huiyi Wang Haodong Lu Lina Yao Dong Gong KELM CLL 45 8 0 27 Mar 2024
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey Zeyu Han Chao Gao Jinyang Liu Jeff Zhang Sai Qian Zhang 150 310 0 21 Mar 2024
DAM: Dynamic Adapter Merging for Continual Video QA Learning Feng Cheng Ziyang Wang Yi-Lin Sung Yan-Bo Lin Mohit Bansal Gedas Bertasius CLL MoMe 36 10 0 13 Mar 2024
RADIN: Souping on a Budget Thibaut Menes Olivier Risser-Maroix MoMe 47 1 0 31 Jan 2024
Multimodal Attention Merging for Improved Speech Recognition and Audio Event Classification Anirudh S. Sundar Chao-Han Huck Yang David M. Chan Shalini Ghosh Venkatesh Ravichandran P. S. Nidadavolu MoMe 40 8 0 22 Dec 2023
Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment Lingling Xu Haoran Xie S. J. Qin Xiaohui Tao F. Wang 49 135 0 19 Dec 2023
Concrete Subspace Learning based Interference Elimination for Multi-task Model Fusion Anke Tang Li Shen Yong Luo Liang Ding Han Hu Bo Du Dacheng Tao MoMe 35 21 0 11 Dec 2023
Customizable Combination of Parameter-Efficient Modules for Multi-Task Learning Haowen Wang Tao Sun Cong Fan Jinjie Gu MoE 16 7 0 06 Dec 2023
Guiding Language Model Math Reasoning with Planning Tokens Xinyi Wang Lucas Caccia O. Ostapenko Xingdi Yuan William Yang Wang Alessandro Sordoni LRM 33 2 0 09 Oct 2023
Parameter Efficient Multi-task Model Fusion with Partial Linearization Anke Tang Li Shen Yong Luo Yibing Zhan Han Hu Bo Du Yixin Chen Dacheng Tao MoMe 26 30 0 07 Oct 2023
TRAM: Bridging Trust Regions and Sharpness Aware Minimization Tom Sherborne Naomi Saphra Pradeep Dasigi Hao Peng 32 4 0 05 Oct 2023
ScaLearn: Simple and Highly Parameter-Efficient Task Transfer by Learning to Scale Markus Frohmann Carolin Holtermann Shahed Masoudian Anne Lauscher Navid Rekabsaz 34 2 0 02 Oct 2023
Deep Model Fusion: A Survey Weishi Li Yong Peng Miao Zhang Liang Ding Han Hu Li Shen FedML MoMe 33 52 0 27 Sep 2023
LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition Chengsong Huang Qian Liu Bill Yuchen Lin Tianyu Pang Chao Du Min-Bin Lin MoMe 38 182 0 25 Jul 2023
Composing Parameter-Efficient Modules with Arithmetic Operations Jinghan Zhang Shiqi Chen Junteng Liu Junxian He KELM MoMe 26 109 0 26 Jun 2023
Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey Chen Ling Xujiang Zhao Jiaying Lu Chengyuan Deng Can Zheng ... Chris White Quanquan Gu Jian Pei Carl Yang Liang Zhao ALM 30 126 0 30 May 2023