Merging Models with Fisher-Weighted Averaging

18 November 2021

Papers citing "Merging Models with Fisher-Weighted Averaging"

50 / 283 papers shown

Title
Robust Concept Erasure Using Task Vectors Minh Pham Kelly O. Marshall Chinmay Hegde Niv Cohen 120 17 0 21 Feb 2025
Sens-Merging: Sensitivity-Guided Parameter Balancing for Merging Large Language Models Shuqi Liu Han Wu Bowei He Xiongwei Han M. Yuan Linqi Song MoMe 63 1 0 20 Feb 2025
Scalable Model Merging with Progressive Layer-wise Distillation Jing Xu Jiazheng Li Junzhe Zhang MoMe FedML 90 0 0 18 Feb 2025
Optimal Brain Iterative Merging: Mitigating Interference in LLM Merging Zhixiang Wang Zhenyu Mao Yixuan Qiao Yunfang Wu Biye Li MoMe 73 0 0 17 Feb 2025
SuperMerge: An Approach For Gradient-Based Model Merging Haoyu Yang Zheng Zhang Saket Sathe MoMe 127 0 0 17 Feb 2025
Forget the Data and Fine-Tuning! Just Fold the Network to Compress Dong Wang Haris Šikić Lothar Thiele O. Saukh 59 0 0 17 Feb 2025
Superpose Singular Features for Model Merging Haiquan Qiu You Wu Quanming Yao MoMe 48 0 0 15 Feb 2025
MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs Yuhang Zhou Giannis Karamanolakis Victor Soto Anna Rumshisky Mayank Kulkarni Furong Huang Wei Ai Jianhua Lu MoMe 106 0 0 03 Feb 2025
Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion Binchi Zhang Zaiyi Zheng Zhengzhang Chen Wenlin Yao 61 0 0 01 Feb 2025
Position: Curvature Matrices Should Be Democratized via Linear Operators Felix Dangel Runa Eschenhagen Weronika Ormaniec Andres Fernandez Lukas Tatzel Agustinus Kristiadi 58 3 0 31 Jan 2025
Reverse Probing: Evaluating Knowledge Transfer via Finetuned Task Embeddings for Coreference Resolution Tatiana Anikina Arne Binder David Harbecke Stalin Varanasi Leonhard Hennig Simon Ostermann Sebastian Möller Josef van Genabith 94 0 0 31 Jan 2025
Task Arithmetic in Trust Region: A Training-Free Model Merging Approach to Navigate Knowledge Conflicts Wenju Sun Qingyong Li Wen Wang Yangli-ao Geng Boyang Li 44 2 0 28 Jan 2025
Evolutionary Optimization of Model Merging Recipes Takuya Akiba Makoto Shing Yujin Tang Qi Sun David Ha MoMe 116 100 0 28 Jan 2025
Multi-Task Model Merging via Adaptive Weight Disentanglement Feng Xiong Runxi Cheng Wang Chen Zhanqiu Zhang Yiwen Guo Chun Yuan Ruifeng Xu MoMe 102 4 0 10 Jan 2025
Soup to go: mitigating forgetting during continual learning with model averaging Anat Kleiman Gintare Karolina Dziugaite Jonathan Frankle Sham Kakade Mansheej Paul MoMe CLL KELM 58 0 0 09 Jan 2025
Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmetic Yifei He Yuzheng Hu Yong Lin Tong Zhang Han Zhao FedML MoMe 65 18 0 08 Jan 2025
Training-free Heterogeneous Model Merging Zhengqi Xu Han Zheng Jie Song Li Sun Mingli Song MoMe 72 1 0 03 Jan 2025
Non-Uniform Parameter-Wise Model Merging Albert Manuel Orozco Camacho Stefan Horoi Guy Wolf Eugene Belilovsky MoMe FedML 90 0 0 20 Dec 2024
Bias Vector: Mitigating Biases in Language Models with Task Arithmetic Approach Daiki Shirafuji Makoto Takenaka Shinya Taguchi LLMAG 72 0 0 16 Dec 2024
SafetyDPO: Scalable Safety Alignment for Text-to-Image Generation Runtao Liu Chen I Chieh Jindong Gu Jipeng Zhang Renjie Pi Qifeng Chen Philip H. S. Torr Ashkan Khakzar Fabio Pizzati EGVM 109 0 0 13 Dec 2024
How to Merge Your Multimodal Models Over Time? Sebastian Dziadzio Vishaal Udandarao Karsten Roth Ameya Prabhu Zeynep Akata Samuel Albanie Matthias Bethge MoMe 98 3 0 09 Dec 2024
Task Arithmetic Through The Lens Of One-Shot Federated Learning Zhixu Tao I. Mason Sanjeev R. Kulkarni Xavier Boix MoMe FedML 84 3 0 27 Nov 2024
Task Singular Vectors: Reducing Task Interference in Model Merging Antonio Andrea Gargiulo Donato Crisostomi Maria Sofia Bucarelli Simone Scardapane Fabrizio Silvestri Emanuele Rodolà MoMe 87 9 0 26 Nov 2024
Beyond Task Vectors: Selective Task Arithmetic Based on Importance Metrics Tian Bowen Lai Songning Wu Jiemin Shuai Zhihao Ge Shiming Yue Yutao MoMe 70 4 0 25 Nov 2024
FREE-Merging: Fourier Transform for Efficient Model Merging Shenghe Zheng Hongzhi Wang MoMe 77 0 0 25 Nov 2024
ATM: Improving Model Merging by Alternating Tuning and Merging Luca Zhou Daniele Solombrino Donato Crisostomi Maria Sofia Bucarelli Fabrizio Silvestri Emanuele Rodolà MoMe 44 5 0 05 Nov 2024
Collective Model Intelligence Requires Compatible Specialization Jyothish Pari Samy Jelassi Pulkit Agrawal MoMe 51 1 0 04 Nov 2024
MoD: A Distribution-Based Approach for Merging Large Language Models Quy-Anh Dang Chris Ngo MoMe VLM 31 0 0 01 Nov 2024
Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging Li Shen Anke Tang Enneng Yang G. Guo Yong Luo Lefei Zhang Xiaochun Cao Bo Du Dacheng Tao MoMe 32 5 0 29 Oct 2024
FuseFL: One-Shot Federated Learning through the Lens of Causality with Progressive Model Fusion Zhenheng Tang Yonggang Zhang Peijie Dong Y. Cheung Amelie Chi Zhou Bo Han Xiaowen Chu FedML MoMe AI4CE 49 6 0 27 Oct 2024
Model merging with SVD to tie the Knots George Stoica Pratik Ramesh B. Ecsedi Leshem Choshen Judy Hoffman MoMe 36 9 0 25 Oct 2024
Closed-form merging of parameter-efficient modules for Federated Continual Learning Riccardo Salami Pietro Buzzega Matteo Mosconi Jacopo Bonato Luigi Sabetta Simone Calderara FedML MoMe CLL 39 2 0 23 Oct 2024
SurgeryV2: Bridging the Gap Between Model Merging and Multi-Task Learning with Deep Representation Surgery Enneng Yang Li Shen Zhenyi Wang G. Guo Xingwei Wang Xiaocun Cao Jie Zhang Dacheng Tao MoMe 37 4 0 18 Oct 2024
Cross-Lingual Auto Evaluation for Assessing Multilingual LLMs Sumanth Doddapaneni Mohammed Safi Ur Rahman Khan Dilip Venkatesh Raj Dabre Anoop Kunchukuttan Mitesh M. Khapra ELM 40 1 0 17 Oct 2024
Mitigating the Backdoor Effect for Multi-Task Model Merging via Safety-Aware Subspace Jinluan Yang Anke Tang Didi Zhu Zhengyu Chen Li Shen Fei Wu MoMe AAML 62 3 0 17 Oct 2024
LoRA Soups: Merging LoRAs for Practical Skill Composition Tasks Akshara Prabhakar Yuanzhi Li Karthik Narasimhan Sham Kakade Eran Malach Samy Jelassi MoMe 36 9 0 16 Oct 2024
Exploring Model Kinship for Merging Large Language Models Yedi Hu Yunzhi Yao N. Zhang Shumin Deng H. Chen MoMe 39 1 0 16 Oct 2024
Extracting and Transferring Abilities For Building Multi-lingual Ability-enhanced Large Language Models Zhipeng Chen Liang Song K. Zhou Wayne Xin Zhao Binghui Wang Weipeng Chen Ji-Rong Wen 68 0 0 10 Oct 2024
Glider: Global and Local Instruction-Driven Expert Router Pingzhi Li Prateek Yadav Jaehong Yoon Jie Peng Yi-Lin Sung Joey Tianyi Zhou Tianlong Chen MoMe MoE 33 1 0 09 Oct 2024
Decouple-Then-Merge: Finetune Diffusion Models as Multi-Task Learning Qianli Ma Xuefei Ning Dongrui Liu Li Niu Linfeng Zhang MoMe 46 0 0 09 Oct 2024
QT-DoG: Quantization-aware Training for Domain Generalization Saqib Javed Hieu Le Mathieu Salzmann OOD MQ 28 1 0 08 Oct 2024
Hyper Adversarial Tuning for Boosting Adversarial Robustness of Pretrained Large Vision Models Kangtao Lv Huangsen Cao Kainan Tu Yihuai Xu Zhimeng Zhang Xin Ding Yongwei Wang MoMe AAML VLM 29 1 0 08 Oct 2024
Wolf2Pack: The AutoFusion Framework for Dynamic Parameter Fusion Bowen Tian Songning Lai Yutao Yue MoMe 30 0 0 08 Oct 2024
Model-GLUE: Democratized LLM Scaling for A Large Model Zoo in the Wild Xinyu Zhao Guoheng Sun Ruisi Cai Yukun Zhou Pingzhi Li ... Binhang Yuan Hongyi Wang Ang Li Zhangyang Wang Tianlong Chen MoMe ALM 28 3 0 07 Oct 2024
Low-Rank Continual Personalization of Diffusion Models Łukasz Staniszewski Katarzyna Zaleska Kamil Deja DiffM 44 0 0 07 Oct 2024
What Matters for Model Merging at Scale? Prateek Yadav Tu Vu Jonathan Lai Alexandra Chronopoulou Manaal Faruqui Joey Tianyi Zhou Tsendsuren Munkhdalai MoMe 46 15 0 04 Oct 2024
Parameter Competition Balancing for Model Merging Guodong Du Junlin Lee Jing Li Runhua Jiang Yifei Guo ... Hanting Liu S. Goh Ho-Kin Tang Daojing He Min Zhang MoMe 35 12 0 03 Oct 2024
DaWin: Training-free Dynamic Weight Interpolation for Robust Adaptation Changdae Oh Yixuan Li Kyungwoo Song Sangdoo Yun Dongyoon Han OOD MoMe 45 4 0 03 Oct 2024
Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Merging Tingfeng Hui Zhenyu Zhang Shuohuan Wang Yu Sun Hua-Hong Wu Sen Su MoE 28 0 0 02 Oct 2024
Foldable SuperNets: Scalable Merging of Transformers with Different Initializations and Tasks Edan Kinderman Itay Hubara Haggai Maron Daniel Soudry MoMe 49 1 0 02 Oct 2024