A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications

10 March 2025

Papers citing "A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications"

50 / 203 papers shown

Title
Second Opinion Matters: Towards Adaptive Clinical AI via the Consensus of Expert Model Ensemble Amit Kumthekar Zion Tilley Henry Duong Bhargav Patel Michael Magnoli Ahmed Omar Ahmed Nasser Chaitanya Gharpure Yevgen Reztzov 34 0 0 29 May 2025
EvidenceMoE: A Physics-Guided Mixture-of-Experts with Evidential Critics for Advancing Fluorescence Light Detection and Ranging in Scattering Media Ismail Erbas Ferhat Demirkiran Karthik Swaminathan Naigang Wang Navid Ibtehaj Nizam Stefan T. Radev Kaoutar El Maghraoui Xavier Intes Vikas Pandey MoE 22 0 0 23 May 2025
MINGLE: Mixtures of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Merging Zihuan Qiu Yi Xu Chiyuan He Fanman Meng Linfeng Xu Qi Wu Hongliang Li CLL MoMe 78 0 0 17 May 2025
Mixture-of-Shape-Experts (MoSE): End-to-End Shape Dictionary Framework to Prompt SAM for Generalizable Medical Segmentation Jia Wei Xiaoqi Zhao Jonghye Woo Georges El Fakhri Xiaofeng Liu Qingyu Chen Xiaofeng Liu 60 0 0 13 Apr 2025
Theory on Mixture-of-Experts in Continual Learning Hongbo Li Sen-Fon Lin Lingjie Duan Yingbin Liang Ness B. Shroff MoE MoMe CLL 254 17 0 20 Feb 2025
Towards Foundational Models for Dynamical System Reconstruction: Hierarchical Meta-Learning via Mixture of Experts Roussel Desmond Nzoyem Grant Stevens Amarpal Sahota David A.W. Barton Tom Deakin 157 3 0 07 Feb 2025
SMOSE: Sparse Mixture of Shallow Experts for Interpretable Reinforcement Learning in Continuous Control Tasks Mátyás Vincze Laura Ferrarotti Leonardo Lucio Custode Bruno Lepri Giovanni Iacca MoE OffRL 120 1 0 17 Dec 2024
Efficient Diffusion Transformer Policies with Mixture of Expert Denoisers for Multitask Learning Moritz Reuss Jyothish Pari Pulkit Agrawal Rudolf Lioutikov DiffM MoE 123 8 0 17 Dec 2024
Towards Adversarial Robustness of Model-Level Mixture-of-Experts Architectures for Semantic Segmentation Svetlana Pavlitska Enrico Eisen J. Marius Zöllner AAML UQCV MoE 141 1 0 16 Dec 2024
Object Detection using Event Camera: A MoE Heat Conduction based Detector and A New Benchmark Dataset Xinyu Wang Yu Jin Wentao Wu Wei Zhang Lin Zhu Bo Jiang Yonghong Tian 87 5 0 09 Dec 2024
Customize Segment Anything Model for Multi-Modal Semantic Segmentation with Mixture of LoRA Experts Chenyang Zhu Bin Xiao Lin Shi Shoukun Xu Xu Zheng MoE 144 12 0 05 Dec 2024
Mixture of Experts in Image Classification: What's the Sweet Spot? Mathurin Videau Alessandro Leite Marc Schoenauer O. Teytaud MoE VLM 78 3 0 27 Nov 2024
LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Training Xiaoye Qu Daize Dong Xuyang Hu Tong Zhu Weigao Sun Yu Cheng MoE 116 13 0 24 Nov 2024
Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent Xingwu Sun Yanfeng Chen Yanwen Huang Ruobing Xie Jiaqi Zhu ... Zhanhui Kang Yong Yang Yuhong Liu Di Wang Jie Jiang MoE ALM ELM 102 33 0 04 Nov 2024
Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging Li Shen Anke Tang Enneng Yang G. Guo Yong Luo Lefei Zhang Xiaochun Cao Di Lin Dacheng Tao MoMe 63 8 0 29 Oct 2024
ViMoE: An Empirical Study of Designing Vision Mixture-of-Experts Xumeng Han Longhui Wei Zhiyang Dou Zipeng Wang Chenhui Qiang Xin He Yingfei Sun Zhenjun Han Qi Tian MoE 61 5 0 21 Oct 2024
MENTOR: Mixture-of-Experts Network with Task-Oriented Perturbation for Visual Reinforcement Learning Suning Huang Zheyu Zhang Tianhai Liang Yihan Xu Zhehao Kou Chenhao Lu Guowei Xu Zhengrong Xue Huazhe Xu MoE 83 4 0 19 Oct 2024
MoH: Multi-Head Attention as Mixture-of-Head Attention Peng Jin Bo Zhu Li Yuan Shuicheng Yan MoE 64 16 0 15 Oct 2024
Reevaluating Meta-Learning Optimization Algorithms Through Contextual Self-Modulation Roussel Desmond Nzoyem David A.W. Barton Tom Deakin 76 4 0 02 Oct 2024
CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcycling Jihai Zhang Xiaoye Qu Tong Zhu Yu Cheng 72 8 0 28 Sep 2024
Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts Xiaoming Shi Shiyu Wang Yuqi Nie Dianqi Li Zhou Ye Qingsong Wen Ming Jin AI4TS 93 44 0 24 Sep 2024
Pluralistic Salient Object Detection Xuelu Feng Yunsheng Li Dongdong Chen Chunming Qiao Junsong Yuan Lu Yuan G. Hua 59 1 0 04 Sep 2024
Jamba-1.5: Hybrid Transformer-Mamba Models at Scale Jamba Team Barak Lenz Alan Arazi Amir Bergman Avshalom Manevich ... Yehoshua Cohen Yonatan Belinkov Y. Globerson Yuval Peleg Levy Y. Shoham 72 32 0 22 Aug 2024
MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routing Hao Zhou Zhijun Wang Shujian Huang Xin Huang Xue Han Junlan Feng Chao Deng Weihua Luo Jiajun Chen CLL MoE 77 6 0 21 Aug 2024
HMoE: Heterogeneous Mixture of Experts for Language Modeling An Wang Xingwu Sun Ruobing Xie Shuaipeng Li Jiaqi Zhu ... J. N. Han Zhanhui Kang Di Wang Naoaki Okazaki Cheng-zhong Xu MoE 75 18 0 20 Aug 2024
PMoE: Progressive Mixture of Experts with Asymmetric Transformer for Continual Learning Min Jae Jung Romain Rouvoy KELM MoE CLL 74 3 0 31 Jul 2024
MaskMoE: Boosting Token-Level Learning via Routing Mask in Mixture-of-Experts Zhenpeng Su Zijia Lin Xue Bai Xing Wu Yizhe Xiong ... Guangyuan Ma Hui Chen Guiguang Ding Wei Zhou Songlin Hu MoE 59 5 0 13 Jul 2024
MoVEInt: Mixture of Variational Experts for Learning Human-Robot Interactions from Demonstrations V. Prasad Alap Kshirsagar Dorothea Koert R. Stock-Homburg Jan Peters Georgia Chalvatzaki DRL 111 6 0 10 Jul 2024
FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs Keyu An Qian Chen Chong Deng Zhihao Du Changfeng Gao ... Bin Zhang Qinglin Zhang Shiliang Zhang Nan Zhao Siqi Zheng AuLLM 89 55 0 04 Jul 2024
Mixture of Experts in a Mixture of RL settings Timon Willi J. Obando-Ceron Jakob Foerster Karolina Dziugaite Pablo Samuel Castro MoE 110 9 0 26 Jun 2024
LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training Tong Zhu Xiaoye Qu Daize Dong Jiacheng Ruan Jingqi Tong Conghui He Yu Cheng MoE ALM 71 82 0 24 Jun 2024
Low-Rank Mixture-of-Experts for Continual Medical Image Segmentation Qian Chen Lei Zhu Hangzhou He Xinliang Zhang Shuang Zeng Qiushi Ren Yanye Lu CLL 79 3 0 19 Jun 2024
Continual Traffic Forecasting via Mixture of Experts Sanghyun Lee Chanyoung Park CLL 82 3 0 05 Jun 2024
Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models Tianwen Wei Bo Zhu Liang Zhao Cheng Cheng Biye Li ... Yutuan Ma Rui Hu Shuicheng Yan Han Fang Yahui Zhou MoE 92 29 0 03 Jun 2024
Yuan 2.0-M32: Mixture of Experts with Attention Router Shaohua Wu Jiangang Luo Xi Chen Lingjun Li Xudong Zhao ... Houbo He Zeru Zhang Zeyu Sun Junxiong Mao Chong Shen MoE 47 9 0 28 May 2024
Decomposing the Neurons: Activation Sparsity via Mixture of Experts for Continual Test Time Adaptation Rongyu Zhang Aosong Cheng Yulin Luo Gaole Dai Huanrui Yang ... Ran Xu Li Du Yuan Du Yanbing Jiang Shanghang Zhang MoE TTA 73 6 0 26 May 2024
Mixture of Experts Meets Prompt-Based Continual Learning Minh Le An Nguyen Huy Nguyen Trang Nguyen Trang Pham L. Ngo Nhat Ho CLL 86 12 0 23 May 2024
Statistical Advantages of Perturbing Cosine Router in Mixture of Experts Huy Le Nguyen Pedram Akbarian Trang Pham Trang Nguyen Shujian Zhang Nhat Ho MoE 84 2 0 23 May 2024
Learning More Generalized Experts by Merging Experts in Mixture-of-Experts Sejik Park FedML CLL MoMe 56 5 0 19 May 2024
Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts Yunxin Li Shenyuan Jiang Baotian Hu Longyue Wang Wanqi Zhong Wenhan Luo Lin Ma Min Zhang MoE 70 34 0 18 May 2024
Swin2-MoSE: A New Single Image Super-Resolution Model for Remote Sensing Leonardo Rossi Vittorio Bernuzzi Tomaso Fontanini Massimo Bertozzi Andrea Prati 77 4 0 29 Apr 2024
M3oE: Multi-Domain Multi-Task Mixture-of Experts Recommendation Framework Zijian Zhang Shuchang Liu Jiaao Yu Qingpeng Cai Xiangyu Zhao ... Qidong Liu Hongwei Zhao Lantao Hu Peng Jiang Kun Gai 68 18 0 29 Apr 2024
MoA: Mixture-of-Attention for Subject-Context Disentanglement in Personalized Image Generation Kuan-Chieh Wang Daniil Ostashev Yuwei Fang Sergey Tulyakov Kfir Aberman 56 24 0 17 Apr 2024
Jamba: A Hybrid Transformer-Mamba Language Model Opher Lieber Barak Lenz Hofit Bata Gal Cohen Jhonathan Osin ... Nir Ratner N. Rozen Erez Shwartz Mor Zusman Y. Shoham 74 219 0 28 Mar 2024
Multi-Task Dense Prediction via Mixture of Low-Rank Experts Yuqi Yang Peng-Tao Jiang Qibin Hou Hao Zhang Jinwei Chen Yue Liu MoE 47 20 0 26 Mar 2024
GeRM: A Generalist Robotic Model with Mixture-of-experts for Quadruped Robot Wenxuan Song Han Zhao Pengxiang Ding Can Cui Shangke Lyu Yaning Fan Donglin Wang OffRL 54 12 0 20 Mar 2024
Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters Jiazuo Yu Yunzhi Zhuge Lu Zhang Ping Hu Dong Wang Huchuan Lu You He VLM KELM CLL OODD 149 81 0 18 Mar 2024
Mixture-of-Prompt-Experts for Multi-modal Semantic Understanding Zichen Wu Hsiu-Yuan Huang Fanyi Qu Hao Sun VLM MoE 69 5 0 17 Mar 2024
Harder Tasks Need More Experts: Dynamic Routing in MoE Models Quzhe Huang Zhenwei An Zhuang Nan Mingxu Tao Chen Zhang ... Kun Xu Kun Xu Liwei Chen Songfang Huang Yansong Feng MoE 56 27 0 12 Mar 2024
Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Experts Onur Celik Aleksandar Taranovic Gerhard Neumann 63 9 0 11 Mar 2024