Mixture-of-Experts with Expert Choice Routing

18 February 2022

Papers citing "Mixture-of-Experts with Expert Choice Routing"

50 / 63 papers shown

Title
UMoE: Unifying Attention and FFN with Shared Experts Yuanhang Yang Chaozheng Wang Jing Li MoE 29 0 0 12 May 2025
Learning Soft Sparse Shapes for Efficient Time-Series Classification Zhen Liu Yicheng Luo B. Li Emadeldeen Eldele Min-man Wu Qianli Ma AI4TS 29 0 0 11 May 2025
Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routing Piotr Piekos Róbert Csordás Jürgen Schmidhuber MoE VLM 94 1 0 01 May 2025
MambaMoE: Mixture-of-Spectral-Spatial-Experts State Space Model for Hyperspectral Image Classification Yichu Xu Di Wang Hongzan Jiao L. Zhang L. Zhang Mamba 83 0 0 29 Apr 2025
Accelerating Mixture-of-Experts Training with Adaptive Expert Replication Athinagoras Skiadopoulos Mark Zhao Swapnil Gandhi Thomas Norrie Shrijeet Mukherjee Christos Kozyrakis MoE 91 0 0 28 Apr 2025
Taming the Titans: A Survey of Efficient LLM Inference Serving Ranran Zhen J. Li Yixin Ji Z. Yang Tong Liu Qingrong Xia Xinyu Duan Z. Wang Baoxing Huai M. Zhang LLMAG 77 0 0 28 Apr 2025
BadMoE: Backdooring Mixture-of-Experts LLMs via Optimizing Routing Triggers and Infecting Dormant Experts Qingyue Wang Qi Pang Xixun Lin Shuai Wang Daoyuan Wu MoE 57 0 0 24 Apr 2025
Dense Backpropagation Improves Training for Sparse Mixture-of-Experts Ashwinee Panda Vatsal Baherwani Zain Sarwar Benjamin Thérien Supriyo Chakraborty Tom Goldstein MoE 37 0 0 16 Apr 2025
Exploiting Mixture-of-Experts Redundancy Unlocks Multimodal Generative Abilities Raman Dutt Harleen Hanspal Guoxuan Xia Petru-Daniel Tudosiu Alexander Black Yongxin Yang Steven G. McDonagh Sarah Parisot MoE 38 0 0 28 Mar 2025
A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications Siyuan Mu Sen Lin MoE 120 1 0 10 Mar 2025
Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts Weigao Sun Disen Lan Tong Zhu Xiaoye Qu Yu-Xi Cheng MoE 97 1 0 07 Mar 2025
Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization Taishi Nakamura Takuya Akiba Kazuki Fujii Yusuke Oda Rio Yokota Jun Suzuki MoMe MoE 81 1 0 26 Feb 2025
CAMEx: Curvature-aware Merging of Experts Dung V. Nguyen Minh H. Nguyen Luc Q. Nguyen R. Teo T. Nguyen Linh Duy Tran MoMe 81 2 0 26 Feb 2025
Filtered not Mixed: Stochastic Filtering-Based Online Gating for Mixture of Large Language Models Raeid Saqur Anastasis Kratsios Florian Krach Yannick Limmer Jacob-Junqi Tian John Willes Blanka Horvath Frank Rudzicz MoE 45 0 0 24 Feb 2025
Tight Clusters Make Specialized Experts Stefan K. Nielsen R. Teo Laziz U. Abdullaev Tan M. Nguyen MoE 61 2 0 21 Feb 2025
Theory on Mixture-of-Experts in Continual Learning Hongbo Li Sen-Fon Lin Lingjie Duan Yingbin Liang Ness B. Shroff MoE MoMe CLL 151 14 0 20 Feb 2025
Importance Sampling via Score-based Generative Models Heasung Kim Taekyun Lee Hyeji Kim Gustavo de Veciana MedIm DiffM 131 1 0 07 Feb 2025
Rank Also Matters: Hierarchical Configuration for Mixture of Adapter Experts in LLM Fine-Tuning Peizhuang Cong Wenpu Liu Wenhan Yu Haochen Zhao Tong Yang ALM MoE 76 0 0 06 Feb 2025
MJ-VIDEO: Fine-Grained Benchmarking and Rewarding Video Preferences in Video Generation Haibo Tong Zhaoyang Wang Z. Chen Haonian Ji Shi Qiu ... Peng Xia Mingyu Ding Rafael Rafailov Chelsea Finn Huaxiu Yao EGVM VGen 95 2 0 03 Feb 2025
Generate to Discriminate: Expert Routing for Continual Learning Yewon Byun Sanket Vaibhav Mehta Saurabh Garg Emma Strubell Michael Oberst Bryan Wilder Zachary Chase Lipton 76 0 0 31 Dec 2024
Mixture of Experts for Node Classification Yu Shi Yiqi Wang WeiXuan Lang Jiaxin Zhang Pan Dong Aiping Li 168 0 0 30 Nov 2024
Complexity Experts are Task-Discriminative Learners for Any Image Restoration Eduard Zamfir Zongwei Wu Nancy Mehta Yuedong Tan Danda Pani Paudel Yulun Zhang Radu Timofte MoE 157 1 0 27 Nov 2024
AnyEdit: Mastering Unified High-Quality Image Editing for Any Idea Qifan Yu Wei Chow Zhongqi Yue Kaihang Pan Yang Wu Xiaoyang Wan Juncheng Billy Li Siliang Tang H. Zhang Yueting Zhuang DiffM 98 15 0 24 Nov 2024
Lifelong Knowledge Editing for Vision Language Models with Low-Rank Mixture-of-Experts Qizhou Chen Chengyu Wang Dakan Wang Taolin Zhang Wangyue Li Xiaofeng He KELM 78 1 0 23 Nov 2024
Mixture of Parrots: Experts improve memorization more than reasoning Samy Jelassi Clara Mohri David Brandfonbrener Alex Gu Nikhil Vyas Nikhil Anand David Alvarez-Melis Yuanzhi Li Sham Kakade Eran Malach MoE 28 4 0 24 Oct 2024
Faster Language Models with Better Multi-Token Prediction Using Tensor Decomposition Artem Basharin Andrei Chertkov Ivan V. Oseledets 40 1 0 23 Oct 2024
ViMoE: An Empirical Study of Designing Vision Mixture-of-Experts Xumeng Han Longhui Wei Zhiyang Dou Zipeng Wang Chenhui Qiang Xin He Yingfei Sun Zhenjun Han Qi Tian MoE 37 3 0 21 Oct 2024
MomentumSMoE: Integrating Momentum into Sparse Mixture of Experts R. Teo Tan M. Nguyen MoE 31 3 0 18 Oct 2024
MoH: Multi-Head Attention as Mixture-of-Head Attention Peng Jin Bo Zhu Li Yuan Shuicheng Yan MoE 29 13 0 15 Oct 2024
Ada-K Routing: Boosting the Efficiency of MoE-based LLMs Tongtian Yue Longteng Guo Jie Cheng Xuange Gao J. Liu MoE 25 0 0 14 Oct 2024
Scalable Multi-Domain Adaptation of Language Models using Modular Experts Peter Schafhalter Shun Liao Yanqi Zhou Chih-Kuan Yeh Arun Kandoor James Laudon MoE 24 1 0 14 Oct 2024
Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Models Jun Luo C. L. P. Chen Shandong Wu FedML VLM MoE 44 3 0 14 Oct 2024
More Experts Than Galaxies: Conditionally-overlapping Experts With Biologically-Inspired Fixed Routing Sagi Shaier Francisco Pereira K. Wense Lawrence E Hunter Matt Jones MoE 46 0 0 10 Oct 2024
EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routing Haotian Sun Tao Lei Bowen Zhang Yanghao Li Haoshuo Huang Ruoming Pang Bo Dai Nan Du DiffM MoE 73 5 0 02 Oct 2024
Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RL Ghada Sokar J. Obando-Ceron Aaron C. Courville Hugo Larochelle Pablo Samuel Castro MoE 114 2 0 02 Oct 2024
Continual learning with the neural tangent ensemble Ari S. Benjamin Christian Pehle Kyle Daruwalla UQCV 60 0 0 30 Aug 2024
Layerwise Recurrent Router for Mixture-of-Experts Zihan Qiu Zeyu Huang Shuang Cheng Yizhi Zhou Zili Wang Ivan Titov Jie Fu MoE 73 2 0 13 Aug 2024
Understanding the Performance and Estimating the Cost of LLM Fine-Tuning Yuchen Xia Jiho Kim Yuhan Chen Haojie Ye Souvik Kundu Cong Hao Nishil Talati MoE 35 19 0 08 Aug 2024
MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts Xi Victoria Lin Akshat Shrivastava Liang Luo Srinivasan Iyer Mike Lewis Gargi Gosh Luke Zettlemoyer Armen Aghajanyan MoE 38 20 0 31 Jul 2024
Stretching Each Dollar: Diffusion Training from Scratch on a Micro-Budget Vikash Sehwag Xianghao Kong Jingtao Li Michael Spranger Lingjuan Lyu DiffM 41 9 0 22 Jul 2024
Node-wise Filtering in Graph Neural Networks: A Mixture of Experts Approach Haoyu Han Juanhui Li Wei Huang Xianfeng Tang Hanqing Lu Chen Luo Hui Liu Jiliang Tang 40 5 0 05 Jun 2024
Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models Yongxin Guo Zhenglin Cheng Xiaoying Tang Tao R. Lin Tao Lin MoE 53 7 0 23 May 2024
SUTRA: Scalable Multilingual Language Model Architecture Abhijit Bendale Michael Sapienza Steven Ripplinger Simon Gibbs Jaewon Lee Pranav Mistry LRM ELM 34 4 0 07 May 2024
Double Mixture: Towards Continual Event Detection from Speech Jingqi Kang Tongtong Wu Jinming Zhao Guitao Wang Yinwei Wei Haomiao Yang Guilin Qi Yuan-Fang Li Gholamreza Haffari 26 0 0 20 Apr 2024
Multimodal Clinical Trial Outcome Prediction with Large Language Models Wenhao Zheng Dongsheng Peng Hongxia Xu Yun-Qing Li Hongtu Zhu Tianfan Fu Huaxiu Yao Huaxiu Yao 42 5 0 09 Feb 2024
Rethinking RGB Color Representation for Image Restoration Models Jaerin Lee J. Park Sungyong Baik Kyoung Mu Lee 19 1 0 05 Feb 2024
FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion Xing Han Huy Nguyen Carl Harris Nhat Ho S. Saria MoE 77 16 0 05 Feb 2024
Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Survey Garima Agrawal Tharindu Kumarage Zeyad Alghami Huanmin Liu 32 81 0 14 Nov 2023
A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts Huy Nguyen Pedram Akbarian TrungTin Nguyen Nhat Ho 29 10 0 22 Oct 2023
Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer Boan Liu Liang Ding Li Shen Keqin Peng Yu Cao Dazhao Cheng Dacheng Tao MoE 36 7 0 15 Oct 2023