Demystifying Softmax Gating Function in Gaussian Mixture of Experts

Demystifying Softmax Gating Function in Gaussian Mixture of Experts

5 May 2023

Huy Nguyen

TrungTin Nguyen

Papers citing "Demystifying Softmax Gating Function in Gaussian Mixture of Experts"

17 / 17 papers shown

Title
A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications Siyuan Mu Sen Lin MoE 147 2 0 10 Mar 2025
Quadratic Gating Functions in Mixture of Experts: A Statistical Insight Pedram Akbarian Huy Le Nguyen Xing Han Nhat Ho MoE 42 0 0 15 Oct 2024
Revisiting Prefix-tuning: Statistical Benefits of Reparameterization among Prompts Minh Le Chau Nguyen Huy Nguyen Quyen Tran Trung Le Nhat Ho 44 4 0 03 Oct 2024
Beyond Parameter Count: Implicit Bias in Soft Mixture of Experts Youngseog Chung Dhruv Malik J. Schneider Yuanzhi Li Aarti Singh MoE 42 1 0 02 Sep 2024
Mixture of Experts Meets Prompt-Based Continual Learning Minh Le An Nguyen Huy Nguyen Trang Nguyen Trang Pham L. Ngo Nhat Ho CLL 49 7 0 23 May 2024
Statistical Advantages of Perturbing Cosine Router in Mixture of Experts Huy Le Nguyen Pedram Akbarian Trang Pham Trang Nguyen Shujian Zhang Nhat Ho MoE 51 2 0 23 May 2024
Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts Huy Nguyen Nhat Ho Alessandro Rinaldo 55 3 0 22 May 2024
MVMoE: Multi-Task Vehicle Routing Solver with Mixture-of-Experts Jianan Zhou Zhiguang Cao Yaoxin Wu Wen Song Yining Ma Jie Zhang Chi Xu 54 19 0 02 May 2024
Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study Jinze Zhao Peihao Wang Zhangyang Wang MoE 23 2 0 26 Mar 2024
On Parameter Estimation in Deviated Gaussian Mixture of Experts Huy Nguyen Khai Nguyen Nhat Ho 44 0 0 07 Feb 2024
On Least Square Estimation in Softmax Gating Mixture of Experts Huy Nguyen Nhat Ho Alessandro Rinaldo 51 13 0 05 Feb 2024
FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion Xing Han Huy Nguyen Carl Harris Nhat Ho S. Saria MoE 77 16 0 05 Feb 2024
Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts? Huy Nguyen Pedram Akbarian Nhat Ho MoE 30 10 0 25 Jan 2024
A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts Huy Nguyen Pedram Akbarian TrungTin Nguyen Nhat Ho 32 10 0 22 Oct 2023
Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts Huy Nguyen Pedram Akbarian Fanqi Yan Nhat Ho MoE 41 16 0 25 Sep 2023
Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts Huy Nguyen TrungTin Nguyen Khai Nguyen Nhat Ho MoE 46 13 0 12 May 2023
Mixture-of-Experts with Expert Choice Routing Yan-Quan Zhou Tao Lei Han-Chu Liu Nan Du Yanping Huang Vincent Zhao Andrew M. Dai Zhifeng Chen Quoc V. Le James Laudon MoE 160 327 0 18 Feb 2022