Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch

6 November 2023

Papers citing "Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch"

50 / 223 papers shown

Title
SurgeryV2: Bridging the Gap Between Model Merging and Multi-Task Learning with Deep Representation Surgery Enneng Yang Li Shen Zhenyi Wang G. Guo Xingwei Wang Xiaocun Cao Jie Zhang Dacheng Tao MoMe 37 4 0 18 Oct 2024
Collaboratively adding new knowledge to an LLM Rhui Dih Lee L. Wynter CLL MoMe 32 0 0 18 Oct 2024
A Unified View of Delta Parameter Editing in Post-Trained Large-Scale Models Qiaoyu Tang Le Yu Bowen Yu Hongyu Lin Keming Lu Yunfan LU Xianpei Han Le Sun MoMe 34 1 0 17 Oct 2024
Unconstrained Model Merging for Enhanced LLM Reasoning Yiming Zhang Baoyi He Shengyu Zhang Yuhao Fu Qi Zhou ... Guanghan Ning Linyi Li Chunlin Ji Fei Wu Hongxia Yang MoMe 39 0 0 17 Oct 2024
On the Role of Attention Heads in Large Language Model Safety Zhenhong Zhou Haiyang Yu Xinghua Zhang Rongwu Xu Fei Huang Kun Wang Yang Liu Junfeng Fang Yongbin Li 59 5 0 17 Oct 2024
Mitigating the Backdoor Effect for Multi-Task Model Merging via Safety-Aware Subspace Jinluan Yang Anke Tang Didi Zhu Zhengyu Chen Li Shen Fei Wu MoMe AAML 62 3 0 17 Oct 2024
LoRA Soups: Merging LoRAs for Practical Skill Composition Tasks Akshara Prabhakar Yuanzhi Li Karthik Narasimhan Sham Kakade Eran Malach Samy Jelassi MoMe 36 9 0 16 Oct 2024
Merge to Learn: Efficiently Adding Skills to Language Models with Model Merging Jacob Morrison Noah A. Smith Hannaneh Hajishirzi Pang Wei Koh Jesse Dodge Pradeep Dasigi KELM MoMe CLL 45 1 0 16 Oct 2024
Exploring Model Kinship for Merging Large Language Models Yedi Hu Yunzhi Yao N. Zhang Shumin Deng H. Chen MoMe 39 1 0 16 Oct 2024
Multi-trait User Simulation with Adaptive Decoding for Conversational Task Assistants Rafael Ferreira David Semedo João Magalhães 31 1 0 16 Oct 2024
Agent Skill Acquisition for Large Language Models via CycleQD So Kuroki Taishi Nakamura Takuya Akiba Yujin Tang MoMe 36 0 0 16 Oct 2024
Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligence Shangbin Feng Zifeng Wang Yike Wang Sayna Ebrahimi Hamid Palangi ... Nathalie Rauschmayr Yejin Choi Yulia Tsvetkov Chen-Yu Lee Tomas Pfister MoMe 37 3 0 15 Oct 2024
DrivingDojo Dataset: Advancing Interactive and Knowledge-Enriched Driving World Model Yuqi Wang Ke Cheng Jiawei He Qitai Wang Hengchen Dai Yuntao Chen Fei Xia Zhaoxiang Zhang VGen 43 8 0 14 Oct 2024
Self-Data Distillation for Recovering Quality in Pruned Large Language Models Vithursan Thangarasa Ganesh Venkatesh Mike Lasby Nish Sinnadurai Sean Lie SyDa 38 1 0 13 Oct 2024
DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Models Wenlong Deng Yize Zhao V. Vakilian Minghui Chen Xiaoxiao Li Christos Thrampoulidis 45 3 0 12 Oct 2024
ELICIT: LLM Augmentation via External In-Context Capability Futing Wang Jianhao Yan Yue Zhang Tao Lin 44 0 0 12 Oct 2024
DeltaDQ: Ultra-High Delta Compression for Fine-Tuned LLMs via Group-wise Dropout and Separate Quantization Yanfeng Jiang Zelan Yang B. Chen Shen Li Yong Li Tao Li MQ 36 0 0 11 Oct 2024
MergePrint: Merge-Resistant Fingerprints for Robust Black-box Ownership Verification of Large Language Models Shojiro Yamabe Tsubasa Takahashi Futa Waseda Koki Wataoka MoMe 86 1 0 11 Oct 2024
Merging in a Bottle: Differentiable Adaptive Merging (DAM) and the Path from Averaging to Automation Thomas Gauthier-Caron Shamane Siriwardhana Elliot Stein Malikeh Ehghaghi Charles Goddard Mark McQuade Jacob Solawetz Maxime Labonne MoMe 36 2 0 10 Oct 2024
SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture Jiayi Han Liang Du Hongwei Du Xiangguo Zhou Yiwen Wu Weibo Zheng Donghong Han CLL MoMe MoE 38 2 0 10 Oct 2024
How Does Vision-Language Adaptation Impact the Safety of Vision Language Models? Seongyun Lee Geewook Kim Jiyeon Kim Hyunji Lee Hoyeon Chang Sue Hyun Park Minjoon Seo 36 0 0 10 Oct 2024
Extracting and Transferring Abilities For Building Multi-lingual Ability-enhanced Large Language Models Zhipeng Chen Liang Song K. Zhou Wayne Xin Zhao Binghui Wang Weipeng Chen Ji-Rong Wen 68 0 0 10 Oct 2024
Efficient Few-shot Learning for Multi-label Classification of Scientific Documents with Many Classes Tim Schopf Alexander Blatzheim Nektarios Machner Florian Matthes VLM 23 1 0 08 Oct 2024
What Matters for Model Merging at Scale? Prateek Yadav Tu Vu Jonathan Lai Alexandra Chronopoulou Manaal Faruqui Joey Tianyi Zhou Tsendsuren Munkhdalai MoMe 46 16 0 04 Oct 2024
Parameter Competition Balancing for Model Merging Guodong Du Junlin Lee Jing Li Runhua Jiang Yifei Guo ... Hanting Liu S. Goh Ho-Kin Tang Daojing He Min Zhang MoMe 37 12 0 03 Oct 2024
Llama SLayer 8B: Shallow Layers Hold the Key to Knowledge Injection Tianrun Chen Zhentao Tan Tao Gong Yue Wu Qi Chu Bin Liu Jieping Ye Nenghai Yu KELM 56 3 0 03 Oct 2024
Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensembling Yuxuan Yao Han Wu Mingyang Liu Sichun Luo Xiongwei Han Jie Liu Zhijiang Guo Linqi Song 58 4 0 03 Oct 2024
DaWin: Training-free Dynamic Weight Interpolation for Robust Adaptation Changdae Oh Yixuan Li Kyungwoo Song Sangdoo Yun Dongyoon Han OOD MoMe 45 4 0 03 Oct 2024
Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Merging Tingfeng Hui Zhenyu Zhang Shuohuan Wang Yu Sun Hua Wu Sen Su MoE 31 0 0 02 Oct 2024
Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Models Lucas Bandarkar Benjamin Muller Pritish Yuvraj Rui Hou Nayan Singhal Hongjiang Lv Bing-Quan Liu KELM LRM MoMe 52 3 0 02 Oct 2024
DLP-LoRA: Efficient Task-Specific LoRA Fusion with a Dynamic, Lightweight Plugin for Large Language Models Yuxuan Zhang Ruizhe Li MoMe 58 0 0 02 Oct 2024
Mitigating Training Imbalance in LLM Fine-Tuning via Selective Parameter Merging Yiming Ju Ziyi Ni Xingrun Xing Zhixiong Zeng hanyu Zhao Siqi Fan Zheng Zhang MoMe 39 2 0 01 Oct 2024
HM3: Heterogeneous Multi-Class Model Merging Stefan Hackmann MoMe 30 0 0 27 Sep 2024
HM3: Hierarchical Multi-Objective Model Merging for Pretrained Models Yu Zhou Xingyu Wu Jibin Wu Liang Feng Kay Chen Tan MoMe 61 0 0 27 Sep 2024
Realistic Evaluation of Model Merging for Compositional Generalization Derek Tam Yash Kant Brian Lester Igor Gilitschenski Colin Raffel MoMe 35 6 0 26 Sep 2024
Prompt Sliders for Fine-Grained Control, Editing and Erasing of Concepts in Diffusion Models Deepak Sridhar Nuno Vasconcelos DiffM 36 0 0 25 Sep 2024
Supervised Fine-Tuning Achieve Rapid Task Adaption Via Alternating Attention Head Activation Patterns Yang Zhao Li Du Xiao Ding Kai Xiong Ting Liu Bing Qin 23 2 0 24 Sep 2024
Obliviate: Neutralizing Task-agnostic Backdoors within the Parameter-efficient Fine-tuning Paradigm Jaehan Kim Minkyoo Song S. Na Seungwon Shin AAML 41 0 0 21 Sep 2024
Recent Advances in Attack and Defense Approaches of Large Language Models Jing Cui Yishi Xu Zhewei Huang Shuchang Zhou Jianbin Jiao Junge Zhang PILM AAML 57 1 0 05 Sep 2024
Interpreting and Improving Large Language Models in Arithmetic Calculation Wei Zhang Chaoqun Wan Yonggang Zhang Yiu-ming Cheung Xinmei Tian Xu Shen Jieping Ye LRM 29 18 0 03 Sep 2024
From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuning Wei Chen Zhen Huang Liang Xie Binbin Lin Houqiang Li ... Deng Cai Yonggang Zhang Wenxiao Wang Xu Shen Jieping Ye 57 6 0 03 Sep 2024
Leveraging Open Knowledge for Advancing Task Expertise in Large Language Models Yuncheng Yang Yulei Qin Tong Wu Zihan Xu Gang Li ... Yuchen Shi Ke Li Xing Sun Jie Yang Yun Gu ALM OffRL MoE 57 0 0 28 Aug 2024
SQL-GEN: Bridging the Dialect Gap for Text-to-SQL Via Synthetic Data And Model Merging Mohammadreza Pourreza Ruoxi Sun Hailong Li Lesly Miculicich Tomas Pfister Sercan Ö. Arik MoMe 40 5 0 22 Aug 2024
SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Models Anke Tang Li Shen Yong Luo Shuai Xie Han Hu Lefei Zhang Bo Du Dacheng Tao MoMe 42 4 0 19 Aug 2024
MergeRepair: An Exploratory Study on Merging Task-Specific Adapters in Code LLMs for Automated Program Repair Meghdad Dehghan Jie JW Wu Fatemeh H. Fard Ali Ouni MoMe 50 2 0 18 Aug 2024
Activated Parameter Locating via Causal Intervention for Model Merging Fanshuang Kong Richong Zhang Ziqiao Wang MoMe 21 1 0 18 Aug 2024
FuseChat: Knowledge Fusion of Chat Models Fanqi Wan Longguang Zhong Ziyi Yang Ruijun Chen Xiaojun Quan ALM KELM MoMe 32 23 0 15 Aug 2024
Extend Model Merging from Fine-Tuned to Pre-Trained Large Language Models via Weight Disentanglement Le Yu Bowen Yu Haiyang Yu Fei Huang Yongbin Li MoMe 35 5 0 06 Aug 2024
MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuning Yupeng Chen Senmiao Wang Zhihang Lin Zhihang Lin Yushun Zhang Tian Ding Ruoyu Sun Ruoyu Sun CLL 80 1 0 30 Jul 2024
Rapid Switching and Multi-Adapter Fusion via Sparse High Rank Adapters Kartikeya Bhardwaj N. Pandey Sweta Priyadarshi Viswanath Ganapathy Rafael Esteves ... Paul N. Whatmough Risheek Garrepalli M. V. Baalen Harris Teague Markus Nagel MoMe 33 0 0 22 Jul 2024