Merging Models with Fisher-Weighted Averaging

18 November 2021

Papers citing "Merging Models with Fisher-Weighted Averaging"

50 / 284 papers shown

Title
Fusion-PSRO: Nash Policy Fusion for Policy Space Response Oracles Jiesong Lian Yucong Huang Chengdong Ma Mingzhi Wang Ying Wen Long Hu Yixue Hao 62 0 0 31 May 2024
Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignment Keming Lu Bowen Yu Fei Huang Yang Fan Runji Lin Chang Zhou MoMe 29 18 0 28 May 2024
Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Models Chia-Yi Hsu Yu-Lin Tsai Chih-Hsun Lin Pin-Yu Chen Chia-Mu Yu Chun-ying Huang 49 32 0 27 May 2024
Ensembling Diffusion Models via Adaptive Feature Aggregation Cong Wang Kuan Tian Yonghang Guan Jun Zhang Zhiwei Jiang Fei Shen Xiao Han 44 5 0 27 May 2024
MiniCache: KV Cache Compression in Depth Dimension for Large Language Models Akide Liu Jing Liu Zizheng Pan Yefei He Gholamreza Haffari Bohan Zhuang MQ 35 30 0 23 May 2024
EMR-Merging: Tuning-Free High-Performance Model Merging Chenyu Huang Peng Ye Tao Chen Tong He Xiangyu Yue Wanli Ouyang MoMe 46 29 0 23 May 2024
Disperse-Then-Merge: Pushing the Limits of Instruction Tuning via Alignment Tax Reduction Tingchen Fu Deng Cai Lemao Liu Shuming Shi Rui Yan MoMe 55 13 0 22 May 2024
MeteoRA: Multiple-tasks Embedded LoRA for Large Language Models Jingwei Xu Junyu Lai Yunpeng Huang MoE MoMe 38 8 0 19 May 2024
Learning More Generalized Experts by Merging Experts in Mixture-of-Experts Sejik Park FedML CLL MoMe 37 5 0 19 May 2024
Towards Modular LLMs by Building and Reusing a Library of LoRAs O. Ostapenko Zhan Su E. Ponti Laurent Charlin Nicolas Le Roux Matheus Pereira Lucas Caccia Alessandro Sordoni MoMe 41 31 0 18 May 2024
A safety realignment framework via subspace-oriented model fusion for large language models Xin Yi Shunfan Zheng Linlin Wang Xiaoling Wang Liang He 60 20 0 15 May 2024
Localizing Task Information for Improved Model Merging and Compression Ke Wang Nikolaos Dimitriadis Guillermo Ortiz-Jimenez Franccois Fleuret Pascal Frossard MoMe 33 46 0 13 May 2024
Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training Zexuan Zhong Mengzhou Xia Danqi Chen Mike Lewis MoE 57 15 0 06 May 2024
Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models Seungone Kim Juyoung Suk Shayne Longpre Bill Yuchen Lin Jamin Shin Sean Welleck Graham Neubig Moontae Lee Kyungjae Lee Minjoon Seo MoMe ALM ELM 51 168 0 02 May 2024
Creative Problem Solving in Large Language and Vision Models -- What Would it Take? Lakshmi Nair Evana Gizzi Jivko Sinapov MLLM 55 2 0 02 May 2024
IMWA: Iterative Model Weight Averaging Benefits Class-Imbalanced Learning Tasks Zitong Huang Ze Chen Bowen Dong Chaoqi Liang Erjin Zhou Wangmeng Zuo MoMe CLL 46 3 0 25 Apr 2024
DynaMMo: Dynamic Model Merging for Efficient Class Incremental Learning for Medical Images Mohammad Areeb Qazi Ibrahim Almakky Anees Ur Rehman Hashmi Santosh Sanjeev Mohammad Yaqub MoMe 37 3 0 22 Apr 2024
MaxFusion: Plug&Play Multi-Modal Generation in Text-to-Image Diffusion Models Nithin Gopalakrishnan Nair Jeya Maria Jose Valanarasu Vishal M. Patel MoMe 33 7 0 15 Apr 2024
Continuous Language Model Interpolation for Dynamic and Controllable Text Generation Sara Kangaslahti David Alvarez-Melis KELM 31 0 0 10 Apr 2024
Lossless and Near-Lossless Compression for Foundation Models Moshik Hershcovitch Leshem Choshen Andrew Wood Ilias Enmouri Peter Chin S. Sundararaman Danny Harnik 49 6 0 05 Apr 2024
Checkpoint Merging via Bayesian Optimization in LLM Pretraining Deyuan Liu Zecheng Wang Bingning Wang Weipeng Chen Chunshan Li Zhiying Tu Dianhui Chu Bo Li Dianbo Sui MoMe 52 15 0 28 Mar 2024
FissionFusion: Fast Geometric Generation and Hierarchical Souping for Medical Image Analysis Santosh Sanjeev Nuren Zhaksylyk Ibrahim Almakky Anees Ur Rehman Hashmi Mohammad Areeb Qazi Mohammad Yaqub 39 3 0 20 Mar 2024
Arcee's MergeKit: A Toolkit for Merging Large Language Models Charles Goddard Shamane Siriwardhana Malikeh Ehghaghi Luke Meyers Vladimir Karpukhin Brian Benedict Mark McQuade Jacob Solawetz MoMe KELM 90 80 0 20 Mar 2024
FedFisher: Leveraging Fisher Information for One-Shot Federated Learning Divyansh Jhunjhunwala Shiqiang Wang Gauri Joshi FedML 28 6 0 19 Mar 2024
MedMerge: Merging Models for Effective Transfer Learning to Medical Imaging Tasks Ibrahim Almakky Santosh Sanjeev Anees Ur Rehman Hashmi Mohammad Areeb Qazi Mohammad Yaqub Mohammad Yaqub FedML MoMe 82 3 0 18 Mar 2024
Fisher Mask Nodes for Language Model Merging Thennal D K Ganesh Nathan Suchithra M S MoMe AI4CE 47 3 0 14 Mar 2024
DAM: Dynamic Adapter Merging for Continual Video QA Learning Feng Cheng Ziyang Wang Yi-Lin Sung Yan-Bo Lin Mohit Bansal Gedas Bertasius CLL MoMe 36 10 0 13 Mar 2024
Unleashing the Power of Meta-tuning for Few-shot Generalization Through Sparse Interpolated Experts Shengzhuang Chen Jihoon Tack Yunqiao Yang Yee Whye Teh Jonathan Richard Schwarz Ying Wei MoE 43 1 0 13 Mar 2024
Here's a Free Lunch: Sanitizing Backdoored Models with Model Merge Ansh Arora Xuanli He Maximilian Mozes Srinibas Swain Mark Dras Qiongkai Xu SILM MoMe AAML 58 12 0 29 Feb 2024
AdaMergeX: Cross-Lingual Transfer with Large Language Models via Adaptive Adapter Merging Yiran Zhao Wenxuan Zhang Huiming Wang Kenji Kawaguchi Lidong Bing MoMe 32 15 0 29 Feb 2024
Training Neural Networks from Scratch with Parallel Low-Rank Adapters Minyoung Huh Brian Cheung Jeremy Bernstein Phillip Isola Pulkit Agrawal 35 10 0 26 Feb 2024
Knowledge Fusion of Chat LLMs: A Preliminary Technical Report Fanqi Wan Ziyi Yang Longguang Zhong Xiaojun Quan Xinting Huang Wei Bi MoMe 27 1 0 25 Feb 2024
Does Combining Parameter-efficient Modules Improve Few-shot Transfer Accuracy? Nader Asadi Mahdi Beitollahi Yasser H. Khalil Yinchuan Li Guojun Zhang Xi Chen MoMe 40 8 0 23 Feb 2024
Model Composition for Multimodal Large Language Models Chi Chen Yiyang Du Zheng Fang Ziyue Wang Fuwen Luo ... Ming Yan Ji Zhang Fei Huang Maosong Sun Yang Liu MoMe 26 3 0 20 Feb 2024
Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic Rishabh Bhardwaj Do Duc Anh Soujanya Poria MoMe 50 36 0 19 Feb 2024
LaCo: Large Language Model Pruning via Layer Collapse Yifei Yang Zouying Cao Hai Zhao 19 52 0 17 Feb 2024
Generalizability of Mixture of Domain-Specific Adapters from the Lens of Signed Weight Directions and its Application to Effective Model Pruning Tuc Nguyen Thai Le MoMe 28 3 0 16 Feb 2024
Towards Safer Large Language Models through Machine Unlearning Zheyuan Liu Guangyao Dou Zhaoxuan Tan Yijun Tian Meng Jiang KELM MU 32 72 0 15 Feb 2024
Learning to Route Among Specialized Experts for Zero-Shot Generalization Mohammed Muqeeth Haokun Liu Yufan Liu Colin Raffel MoMe 37 34 0 08 Feb 2024
Bayesian Uncertainty for Gradient Aggregation in Multi-Task Learning Idan Achituve I. Diamant Arnon Netzer Gal Chechik Ethan Fetaya UQCV 32 4 0 06 Feb 2024
On the Emergence of Cross-Task Linearity in the Pretraining-Finetuning Paradigm Zhanpeng Zhou Zijun Chen Yilan Chen Bo-Wen Zhang Junchi Yan MoMe 19 9 0 06 Feb 2024
Representation Surgery for Multi-Task Model Merging Enneng Yang Li Shen Zhenyi Wang Guibing Guo Xiaojun Chen Xingwei Wang Dacheng Tao MoMe 56 38 0 05 Feb 2024
Merging Multi-Task Models via Weight-Ensembling Mixture of Experts Anke Tang Li Shen Yong Luo Nan Yin Lefei Zhang Dacheng Tao MoMe 36 41 0 01 Feb 2024
FreeStyle: Free Lunch for Text-guided Style Transfer using Diffusion Models Feihong He Gang Li Mengyuan Zhang Leilei Yan Hui Xiong Fanzhang Li Li Shen DiffM 23 15 0 28 Jan 2024
Learning under Label Noise through Few-Shot Human-in-the-Loop Refinement Aaqib Saeed Dimitris Spathis Jungwoo Oh Edward Choi Ali Etemad NoLa 31 2 0 25 Jan 2024
LLM Augmented LLMs: Expanding Capabilities through Composition Rachit Bansal Bidisha Samanta Siddharth Dalmia Nitish Gupta Shikhar Vashishth Sriram Ganapathy Abhishek Bapna Prateek Jain Partha P. Talukdar CLL 21 34 0 04 Jan 2024
Merging Vision Transformers from Different Tasks and Domains Peng Ye Chenyu Huang Mingzhu Shen Tao Chen Yongqi Huang Yuning Zhang Wanli Ouyang MoMe 22 11 0 25 Dec 2023
Multimodal Attention Merging for Improved Speech Recognition and Audio Event Classification Anirudh S. Sundar Chao-Han Huck Yang David M. Chan Shalini Ghosh Venkatesh Ravichandran P. S. Nidadavolu MoMe 40 8 0 22 Dec 2023
Weighted Ensemble Models Are Strong Continual Learners Imad Eddine Marouf Subhankar Roy Enzo Tartaglione Stéphane Lathuilière CLL 29 16 0 14 Dec 2023
Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Masks Mohammad-Javad Davari Eugene Belilovsky MoMe 42 56 0 11 Dec 2023