MoORE: SVD-based Model MoE-ization for Conflict- and Oblivion-Resistant Multi-Task Adaptation

MoORE: SVD-based Model MoE-ization for Conflict- and Oblivion-Resistant Multi-Task Adaptation

17 June 2025

ArXiv (abs)PDF HTML

Papers citing "MoORE: SVD-based Model MoE-ization for Conflict- and Oblivion-Resistant Multi-Task Adaptation"

19 / 19 papers shown

Title
Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization Taishi Nakamura Takuya Akiba Kazuki Fujii Yusuke Oda Rio Yokota Jun Suzuki MoMe MoE 120 2 0 26 Feb 2025
OMoE: Diversifying Mixture of Low-Rank Adaptation by Orthogonal Finetuning Jinyuan Feng Zhiqiang Pu Tianyi Hu Dongmin Li Xiaolin Ai Huimu Wang MoE 68 5 0 20 Jan 2025
MTL-LoRA: Low-Rank Adaptation for Multi-Task Learning Yaming Yang Dilxat Muhtar Yelong Shen Yuefeng Zhan Jianfeng Liu ... Denvy Deng Feng Sun Qi Zhang Weizhu Chen Yunhai Tong MoE MoMe 124 4 0 12 Oct 2024
Upcycling Large Language Models into Mixture of Experts Ethan He Abhinav Khattar R. Prenger V. Korthikanti Zijie Yan Tong Liu Shiqing Fan Ashwath Aithal Mohammad Shoeybi Bryan Catanzaro MoE 83 16 0 10 Oct 2024
EMR-Merging: Tuning-Free High-Performance Model Merging Chenyu Huang Peng Ye Tao Chen Tong He Xiangyu Yue Wanli Ouyang MoMe 78 45 0 23 May 2024
MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Models Tongxu Luo Jiahe Lei Fangyu Lei Weihao Liu Shizhu He Jun Zhao Kang Liu MoE ALM 77 27 0 20 Feb 2024
Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch Le Yu Yu Bowen Haiyang Yu Fei Huang Yongbin Li MoMe 107 331 0 06 Nov 2023
When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications Qidong Liu Xian Wu Xiangyu Zhao Yuanshao Zhu Derong Xu Feng Tian Yefeng Zheng MoE 79 72 0 21 Oct 2023
Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints Aran Komatsuzaki J. Puigcerver James Lee-Thorp Carlos Riquelme Ruiz Basil Mustafa Joshua Ainslie Yi Tay Mostafa Dehghani N. Houlsby MoMe MoE 76 123 0 09 Dec 2022
Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them Mirac Suzgun Nathan Scales Nathanael Scharli Sebastian Gehrmann Yi Tay ... Aakanksha Chowdhery Quoc V. Le Ed H. Chi Denny Zhou Jason W. Wei ALM ELM LRM ReLM 263 1,131 0 17 Oct 2022
Efficient Large Scale Language Modeling with Mixtures of Experts Mikel Artetxe Shruti Bhosale Naman Goyal Todor Mihaylov Myle Ott ... Jeff Wang Luke Zettlemoyer Mona T. Diab Zornitsa Kozareva Ves Stoyanov MoE 194 198 0 20 Dec 2021
Program Synthesis with Large Language Models Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski ... Ellen Jiang Carrie J. Cai Michael Terry Quoc V. Le Charles Sutton ELM AIMat ReCod ALM 200 1,986 0 16 Aug 2021
Evaluating Large Language Models Trained on Code Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Pondé ... Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever Wojciech Zaremba ELM ALM 233 5,635 0 07 Jul 2021
Scaling Vision with Sparse Mixture of Experts C. Riquelme J. Puigcerver Basil Mustafa Maxim Neumann Rodolphe Jenatton André Susano Pinto Daniel Keysers N. Houlsby MoE 112 606 0 10 Jun 2021
Aligning AI With Shared Human Values Dan Hendrycks Collin Burns Steven Basart Andrew Critch Jingkai Li Basel Alomair Jacob Steinhardt 145 569 0 05 Aug 2020
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions Christopher Clark Kenton Lee Ming-Wei Chang Tom Kwiatkowski Michael Collins Kristina Toutanova 230 1,549 0 24 May 2019
Characterizing and Avoiding Negative Transfer Zirui Wang Zihang Dai Barnabás Póczós J. Carbonell 85 416 0 24 Nov 2018
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 1.1K 7,182 0 20 Apr 2018
A Survey on Multi-Task Learning Yu Zhang Qiang Yang AIMat 605 2,235 0 25 Jul 2017