Fusing finetuned models for better pretraining

6 April 2022

Papers citing "Fusing finetuned models for better pretraining"

50 / 80 papers shown

Title
Mergenetic: a Simple Evolutionary Model Merging Library Adrian Robert Minut Tommaso Mencattini Andrea Santilli Donato Crisostomi Emanuele Rodolà MoMe 19 0 0 16 May 2025
Investigating Task Arithmetic for Zero-Shot Information Retrieval Marco Braga Pranav Kasela Alessandro Raganato G. Pasi RALM 66 0 0 01 May 2025
Ensemble Learning for Large Language Models in Text and Code Generation: A Survey Mari Ashiga Wei Jie Fan Wu Vardan K. Voskanyan Fateme Dinmohammadi P. Brookes Jingzhi Gong Zheng Wang 44 0 0 13 Mar 2025
From Task-Specific Models to Unified Systems: A Review of Model Merging Approaches Wei Ruan Tianze Yang Yue Zhou Tianming Liu Jin Lu MoMe 90 0 0 13 Mar 2025
SplatPose: Geometry-Aware 6-DoF Pose Estimation from Single RGB Image via 3D Gaussian Splatting Linqi Yang Xiongwei Zhao Qihao Sun Ke Wang Ao Chen Peng Kang 3DGS 80 2 0 07 Mar 2025
LEWIS (LayEr WIse Sparsity) -- A Training Free Guided Model Merging Approach Hetarth Chopra Vidhi Rambhia Vikram Adve MoMe 70 0 0 05 Mar 2025
LED-Merging: Mitigating Safety-Utility Conflicts in Model Merging with Location-Election-Disjoint Qianli Ma Dongrui Liu Qian Chen Linfeng Zhang Jing Shao MoMe 147 0 0 24 Feb 2025
LoRE-Merging: Exploring Low-Rank Estimation For Large Language Model Merging Zehua Liu Han Wu Yuxuan Yao Ruifeng She Xiongwei Han Tao Zhong M. Yuan MoMe 44 1 0 15 Feb 2025
Superpose Singular Features for Model Merging Haiquan Qiu You Wu Quanming Yao MoMe 48 0 0 15 Feb 2025
Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging Pierre Ablin Angelos Katharopoulos Skyler Seto David Grangier MoMe 50 0 0 03 Feb 2025
Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion Binchi Zhang Zaiyi Zheng Zhengzhang Chen Wenlin Yao 61 0 0 01 Feb 2025
Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmetic Yifei He Yuzheng Hu Yong Lin Tong Zhang Han Zhao FedML MoMe 65 18 0 08 Jan 2025
SafetyDPO: Scalable Safety Alignment for Text-to-Image Generation Runtao Liu Chen I Chieh Jindong Gu Jipeng Zhang Renjie Pi Qifeng Chen Philip H. S. Torr Ashkan Khakzar Fabio Pizzati EGVM 109 0 0 13 Dec 2024
Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusion Zhuokun Chen Jinwu Hu Zeshuai Deng Yufeng Wang Bohan Zhuang Mingkui Tan 71 0 0 02 Dec 2024
ATM: Improving Model Merging by Alternating Tuning and Merging Luca Zhou Daniele Solombrino Donato Crisostomi Maria Sofia Bucarelli Fabrizio Silvestri Emanuele Rodolà MoMe 47 5 0 05 Nov 2024
Model merging with SVD to tie the Knots George Stoica Pratik Ramesh B. Ecsedi Leshem Choshen Judy Hoffman MoMe 36 9 0 25 Oct 2024
Tracking Universal Features Through Fine-Tuning and Model Merging Niels Horn Desmond Elliott MoMe 33 0 0 16 Oct 2024
Glider: Global and Local Instruction-Driven Expert Router Pingzhi Li Prateek Yadav Jaehong Yoon Jie Peng Yi-Lin Sung Joey Tianyi Zhou Tianlong Chen MoMe MoE 33 1 0 09 Oct 2024
QT-DoG: Quantization-aware Training for Domain Generalization Saqib Javed Hieu Le Mathieu Salzmann OOD MQ 28 1 0 08 Oct 2024
What Matters for Model Merging at Scale? Prateek Yadav Tu Vu Jonathan Lai Alexandra Chronopoulou Manaal Faruqui Joey Tianyi Zhou Tsendsuren Munkhdalai MoMe 46 15 0 04 Oct 2024
Parameter Competition Balancing for Model Merging Guodong Du Junlin Lee Jing Li Runhua Jiang Yifei Guo ... Hanting Liu S. Goh Ho-Kin Tang Daojing He Min Zhang MoMe 35 12 0 03 Oct 2024
Foldable SuperNets: Scalable Merging of Transformers with Different Initializations and Tasks Edan Kinderman Itay Hubara Haggai Maron Daniel Soudry MoMe 49 1 0 02 Oct 2024
Realistic Evaluation of Model Merging for Compositional Generalization Derek Tam Yash Kant Brian Lester Igor Gilitschenski Colin Raffel MoMe 35 6 0 26 Sep 2024
Acceptable Use Policies for Foundation Models Kevin Klyman 31 14 0 29 Aug 2024
SwiftBrush v2: Make Your One-step Diffusion Model Better Than Its Teacher T. Dao Thuan Hoang Nguyen T. Le D. Vu Khoi Nguyen Cuong Pham Anh Tran DiffM 44 13 0 26 Aug 2024
Mitigating Catastrophic Forgetting in Language Transfer via Model Merging Anton Alexandrov Veselin Raychev Mark Niklas Muller Ce Zhang Martin Vechev Kristina Toutanova MoMe CLL KELM 42 13 0 11 Jul 2024
Foundation Model Engineering: Engineering Foundation Models Just as Engineering Software Dezhi Ran Mengzhou Wu Wei Yang Tao Xie AI4CE 39 1 0 11 Jul 2024
Unlocking the Potential of Model Merging for Low-Resource Languages Mingxu Tao Chen Zhang Quzhe Huang Tianyao Ma Songfang Huang Dongyan Zhao Yansong Feng CLL MoMe 27 3 0 04 Jul 2024
Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead Rickard Brüel-Gabrielsson Jiacheng Zhu Onkar Bhardwaj Leshem Choshen Kristjan Greenewald Mikhail Yurochkin Justin Solomon 45 5 0 17 Jun 2024
Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Efforts Zhaoxuan Tan Zheyuan Liu Meng Jiang 36 20 0 15 Jun 2024
Diffusion Soup: Model Merging for Text-to-Image Diffusion Models Benjamin Biggs Arjun Seshadri Yang Zou Achin Jain Aditya Golatkar Yusheng Xie Alessandro Achille Ashwin Swaminathan Stefano Soatto MoMe DiffM 40 10 0 12 Jun 2024
Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Models Chia-Yi Hsu Yu-Lin Tsai Chih-Hsun Lin Pin-Yu Chen Chia-Mu Yu Chun-ying Huang 49 32 0 27 May 2024
Learning More Generalized Experts by Merging Experts in Mixture-of-Experts Sejik Park FedML CLL MoMe 34 5 0 19 May 2024
A Federated Learning Approach to Privacy Preserving Offensive Language Identification Marcos Zampieri Damith Premasiri Tharindu Ranasinghe FedML 24 2 0 17 Apr 2024
Lossless and Near-Lossless Compression for Foundation Models Moshik Hershcovitch Leshem Choshen Andrew Wood Ilias Enmouri Peter Chin S. Sundararaman Danny Harnik 49 6 0 05 Apr 2024
Arcee's MergeKit: A Toolkit for Merging Large Language Models Charles Goddard Shamane Siriwardhana Malikeh Ehghaghi Luke Meyers Vladimir Karpukhin Brian Benedict Mark McQuade Jacob Solawetz MoMe KELM 90 80 0 20 Mar 2024
FedFisher: Leveraging Fisher Information for One-Shot Federated Learning Divyansh Jhunjhunwala Shiqiang Wang Gauri Joshi FedML 28 6 0 19 Mar 2024
Fisher Mask Nodes for Language Model Merging Thennal D K Ganesh Nathan Suchithra M S MoMe AI4CE 47 3 0 14 Mar 2024
Here's a Free Lunch: Sanitizing Backdoored Models with Model Merge Ansh Arora Xuanli He Maximilian Mozes Srinibas Swain Mark Dras Qiongkai Xu SILM MoMe AAML 58 12 0 29 Feb 2024
Does Combining Parameter-efficient Modules Improve Few-shot Transfer Accuracy? Nader Asadi Mahdi Beitollahi Yasser H. Khalil Yinchuan Li Guojun Zhang Xi Chen MoMe 40 8 0 23 Feb 2024
Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic Rishabh Bhardwaj Do Duc Anh Soujanya Poria MoMe 50 36 0 19 Feb 2024
BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains Yanis Labrak Adrien Bazoge Emmanuel Morin P. Gourraud Mickael Rouvier Richard Dufour 105 191 0 15 Feb 2024
Learning to Route Among Specialized Experts for Zero-Shot Generalization Mohammed Muqeeth Haokun Liu Yufan Liu Colin Raffel MoMe 37 34 0 08 Feb 2024
Multimodal Attention Merging for Improved Speech Recognition and Audio Event Classification Anirudh S. Sundar Chao-Han Huck Yang David M. Chan Shalini Ghosh Venkatesh Ravichandran P. S. Nidadavolu MoMe 40 8 0 22 Dec 2023
Weighted Ensemble Models Are Strong Continual Learners Imad Eddine Marouf Subhankar Roy Enzo Tartaglione Stéphane Lathuilière CLL 29 16 0 14 Dec 2023
Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Masks Mohammad-Javad Davari Eugene Belilovsky MoMe 40 56 0 11 Dec 2023
Merging by Matching Models in Task Parameter Subspaces Derek Tam Mohit Bansal Colin Raffel MoMe 21 10 0 07 Dec 2023
ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization Prateek Yadav Leshem Choshen Colin Raffel Mohit Bansal 32 13 0 22 Nov 2023
Fuse to Forget: Bias Reduction and Selective Memorization through Model Fusion Kerem Zaman Leshem Choshen Shashank Srivastava KELM MoMe 25 10 0 13 Nov 2023
SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understanding Haoxiang Wang Pavan Kumar Anasosalu Vasu Fartash Faghri Raviteja Vemulapalli Mehrdad Farajtabar Sachin Mehta Mohammad Rastegari Oncel Tuzel Hadi Pouransari VLM 32 67 0 23 Oct 2023