v1v2v3 (latest)

Arcee's MergeKit: A Toolkit for Merging Large Language Models

20 March 2024

Papers citing "Arcee's MergeKit: A Toolkit for Merging Large Language Models"

50 / 112 papers shown

Title
A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learning Prateek Yadav Colin Raffel Mohammed Muqeeth Lucas Caccia Haokun Liu Tianlong Chen Joey Tianyi Zhou Leshem Choshen Alessandro Sordoni MoMe 97 24 0 13 Aug 2024
Extend Model Merging from Fine-Tuned to Pre-Trained Large Language Models via Weight Disentanglement Le Yu Bowen Yu Haiyang Yu Fei Huang Yongbin Li MoMe 73 8 0 06 Aug 2024
Computer Audition: From Task-Specific Machine Learning to Foundation Models Andreas Triantafyllopoulos Iosif Tsangko Alexander Gebhard A. Mesaros Tuomas Virtanen Björn Schuller 87 4 0 22 Jul 2024
Knowledge Mechanisms in Large Language Models: A Survey and Perspective Meng Wang Yunzhi Yao Ziwen Xu Shuofei Qiao Shumin Deng ... Yong Jiang Pengjun Xie Fei Huang Huajun Chen Ningyu Zhang 101 38 0 22 Jul 2024
Mitigating Catastrophic Forgetting in Language Transfer via Model Merging Anton Alexandrov Veselin Raychev Mark Niklas Muller Ce Zhang Martin Vechev Kristina Toutanova MoMe CLL KELM 77 20 0 11 Jul 2024
It's Morphing Time: Unleashing the Potential of Multiple LLMs via Multi-objective Optimization Bingdong Li Zixiang Di Yanting Yang Hong Qian Peng Yang Hao Hao Ke Tang Aimin Zhou MoMe 97 6 0 29 Jun 2024
WARP: On the Benefits of Weight Averaged Rewarded Policies Alexandre Ramé Johan Ferret Nino Vieillard Robert Dadashi Léonard Hussenot Pierre-Louis Cedoz Pier Giuseppe Sessa Sertan Girgin Arthur Douillard Olivier Bachem 106 21 0 24 Jun 2024
Domain Adaptation of Llama3-70B-Instruct through Continual Pre-Training and Model Merging: A Comprehensive Evaluation Shamane Siriwardhana Mark McQuade Thomas Gauthier Lucas Atkins Fernando Fernandes Neto ... Anneketh Vij Tyler Odenthal Charles Goddard Mary MacCarthy Jacob Solawetz CLL MoMe ALM 56 9 0 21 Jun 2024
LLM-based speaker diarization correction: A generalizable approach Georgios Efstathiadis Vijay Yadav Anzar Abbas 92 3 0 07 Jun 2024
FusionBench: A Comprehensive Benchmark of Deep Model Fusion Anke Tang Li Shen Yong Luo Han Hu Di Lin Dacheng Tao ELM MoMe VLM 66 27 0 05 Jun 2024
Exploring the LLM Journey from Cognition to Expression with Linear Representations Yuzi Yan J. Li Yipin Zhang Dong Yan 64 2 0 27 May 2024
ConStat: Performance-Based Contamination Detection in Large Language Models Jasper Dekoninck Mark Niklas Muller Martin Vechev 39 8 0 25 May 2024
WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models Peng Wang Zexi Li Ningyu Zhang Ziwen Xu Yunzhi Yao Yong Jiang Pengjun Xie Fei Huang Huajun Chen KELM CLL 108 33 0 23 May 2024
EMR-Merging: Tuning-Free High-Performance Model Merging Chenyu Huang Peng Ye Tao Chen Tong He Xiangyu Yue Wanli Ouyang MoMe 78 45 0 23 May 2024
Aloe: A Family of Fine-tuned Open Healthcare LLMs Ashwin Kumar Gururajan Enrique Lopez-Cuena Jordi Bayarri-Planas Adrián Tormos Daniel Hinjos ... Lucia Urcelay-Ganzabal Marta Gonzalez-Mallo Sergio Alvarez-Napagao Eduard Ayguadé-Parra Ulises Cortés Dario Garcia-Gasulla ELM LM&MA 76 15 0 03 May 2024
Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models Seungone Kim Juyoung Suk Shayne Longpre Bill Yuchen Lin Jamin Shin Sean Welleck Graham Neubig Moontae Lee Kyungjae Lee Minjoon Seo MoMe ALM ELM 108 205 0 02 May 2024
Model Extrapolation Expedites Alignment Chujie Zheng Ziqi Wang Heng Ji Minlie Huang Nanyun Peng MoMe 44 33 0 25 Apr 2024
Stepwise Alignment for Constrained Language Model Policy Optimization Akifumi Wachi Thien Q. Tran Rei Sato Takumi Tanabe Yohei Akimoto 68 10 0 17 Apr 2024
Have You Merged My Model? On The Robustness of Large Language Model IP Protection Methods Against Model Merging Tianshuo Cong Delong Ran Zesen Liu Xinlei He Jinyuan Liu Yichen Gong Qi Li Anyu Wang Xiaoyun Wang MoMe 57 8 0 08 Apr 2024
PhyloLM : Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarks Nicolas Yax Pierre-Yves Oudeyer Stefano Palminteri 92 6 0 06 Apr 2024
Simple and Scalable Strategies to Continually Pre-train Large Language Models Adam Ibrahim Benjamin Thérien Kshitij Gupta Mats L. Richter Quentin Anthony Timothée Lesort Eugene Belilovsky Irina Rish KELM CLL 87 60 0 13 Mar 2024
To Generate or to Retrieve? On the Effectiveness of Artificial Contexts for Medical Open-Domain Question Answering Giacomo Frisoni Alessio Cocchieri Alex Presepi Gianluca Moro Zaiqiao Meng RALM MedIm 90 17 0 04 Mar 2024
Merging Text Transformer Models from Different Initializations Neha Verma Maha Elbayad MoMe 102 8 0 01 Mar 2024
OPDAI at SemEval-2024 Task 6: Small LLMs can Accelerate Hallucination Detection with Weakly Supervised Data Chengcheng Wei Ze Chen Songtan Fang Jiarong He Max Gao 48 3 0 20 Feb 2024
BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains Yanis Labrak Adrien Bazoge Emmanuel Morin P. Gourraud Mickael Rouvier Richard Dufour 166 223 0 15 Feb 2024
Institutional Platform for Secure Self-Service Large Language Model Exploration V. Bumgardner Mitchell A. Klusty W. V. Logan Samuel E. Armstrong Caylin D. Hickey Jeff Talbert Caylin Hickey Jeff Talbert 122 1 0 01 Feb 2024
LLaMA Pro: Progressive LLaMA with Block Expansion Chengyue Wu Yukang Gan Yixiao Ge Zeyu Lu Jiahao Wang Ye Feng Ying Shan Ping Luo CLL 79 72 0 04 Jan 2024
LLM Augmented LLMs: Expanding Capabilities through Composition Rachit Bansal Bidisha Samanta Siddharth Dalmia Nitish Gupta Shikhar Vashishth Sriram Ganapathy Abhishek Bapna Prateek Jain Partha P. Talukdar CLL 64 38 0 04 Jan 2024
TinyLlama: An Open-Source Small Language Model Peiyuan Zhang Guangtao Zeng Tianduo Wang Wei Lu ALM LRM 145 406 0 04 Jan 2024
SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling Dahyun Kim Chanjun Park Sanghoon Kim Wonsung Lee Wonho Song ... Hyunbyung Park Gyoungjin Gim Mikyoung Cha Hwalsuk Lee Sunghun Kim ALM ELM 90 148 0 23 Dec 2023
Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Masks Mohammad-Javad Davari Eugene Belilovsky MoMe 78 69 0 11 Dec 2023
Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch Le Yu Yu Bowen Haiyang Yu Fei Huang Yongbin Li MoMe 107 333 0 06 Nov 2023
Adapting Large Language Models via Reading Comprehension Daixuan Cheng Shaohan Huang Furu Wei CLL SyDa AI4CE 61 36 0 18 Sep 2023
Code Llama: Open Foundation Models for Code Baptiste Rozière Jonas Gehring Fabian Gloeckle Sten Sootla Itai Gat ... Hugo Touvron Louis Martin Nicolas Usunier Thomas Scialom Gabriel Synnaeve ELM ALM 119 2,077 0 24 Aug 2023
Llama 2: Open Foundation and Fine-Tuned Chat Models Hugo Touvron Louis Martin Kevin R. Stone Peter Albert Amjad Almahairi ... Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov Thomas Scialom AI4MH ALM 394 12,044 0 18 Jul 2023
Direct Preference Optimization: Your Language Model is Secretly a Reward Model Rafael Rafailov Archit Sharma E. Mitchell Stefano Ermon Christopher D. Manning Chelsea Finn ALM 389 4,139 0 29 May 2023
ZipIt! Merging Models from Different Tasks without Training George Stoica Daniel Bolya J. Bjorner Pratik Ramesh Taylor N. Hearn Judy Hoffman VLM MoMe 106 125 0 04 May 2023
Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalization Alexandre Ramé Kartik Ahuja Jianyu Zhang Matthieu Cord Léon Bottou David Lopez-Paz MoMe OODD 96 85 0 20 Dec 2022
Dataless Knowledge Fusion by Merging Weights of Language Models Xisen Jin Xiang Ren Daniel Preoţiuc-Pietro Pengxiang Cheng FedML MoMe 75 249 0 19 Dec 2022
Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints Aran Komatsuzaki J. Puigcerver James Lee-Thorp Carlos Riquelme Ruiz Basil Mustafa Joshua Ainslie Yi Tay Mostafa Dehghani N. Houlsby MoMe MoE 76 123 0 09 Dec 2022
MegaBlocks: Efficient Sparse Training with Mixture-of-Experts Trevor Gale Deepak Narayanan C. Young Matei A. Zaharia MoE 76 108 0 29 Nov 2022
REPAIR: REnormalizing Permuted Activations for Interpolation Repair Keller Jordan Hanie Sedghi O. Saukh R. Entezari Behnam Neyshabur MoMe 82 101 0 15 Nov 2022
Git Re-Basin: Merging Models modulo Permutation Symmetries Samuel K. Ainsworth J. Hayase S. Srinivasa MoMe 298 343 0 11 Sep 2022
Fusing finetuned models for better pretraining Leshem Choshen Elad Venezian Noam Slonim Yoav Katz FedML AI4CE MoMe 117 96 0 06 Apr 2022
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time Mitchell Wortsman Gabriel Ilharco S. Gadre Rebecca Roelofs Raphael Gontijo-Lopes ... Hongseok Namkoong Ali Farhadi Y. Carmon Simon Kornblith Ludwig Schmidt MoMe 159 1,009 1 10 Mar 2022
Merging Models with Fisher-Weighted Averaging Michael Matena Colin Raffel FedML MoMe 87 402 0 18 Nov 2021
Training Verifiers to Solve Math Word Problems K. Cobbe V. Kosaraju Mohammad Bavarian Mark Chen Heewoo Jun ... Jerry Tworek Jacob Hilton Reiichiro Nakano Christopher Hesse John Schulman ReLM OffRL LRM 342 4,569 0 27 Oct 2021
Multitask Prompted Training Enables Zero-Shot Task Generalization Victor Sanh Albert Webson Colin Raffel Stephen H. Bach Lintang Sutawika ... T. Bers Stella Biderman Leo Gao Thomas Wolf Alexander M. Rush LRM 355 1,708 0 15 Oct 2021
The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks R. Entezari Hanie Sedghi O. Saukh Behnam Neyshabur MoMe 94 237 0 12 Oct 2021
TruthfulQA: Measuring How Models Mimic Human Falsehoods Stephanie C. Lin Jacob Hilton Owain Evans HILM 149 1,938 0 08 Sep 2021