StableMoE: Stable Routing Strategy for Mixture of Experts

18 April 2022

Zhifang Sui

Papers citing "StableMoE: Stable Routing Strategy for Mixture of Experts"

22 / 22 papers shown

Title
DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs Minxuan Lv Zhenpeng Su Leiyu Pan Yizhe Xiong Zijia Lin ... Guiguang Ding Cheng Luo Di Zhang Kun Gai Songlin Hu MoE 97 0 0 18 Feb 2025
Importance Sampling via Score-based Generative Models Heasung Kim Taekyun Lee Hyeji Kim Gustavo de Veciana MedIm DiffM 188 2 0 07 Feb 2025
Layerwise Recurrent Router for Mixture-of-Experts Zihan Qiu Zeyu Huang Shuang Cheng Yizhi Zhou Zili Wang Ivan Titov Jie Fu MoE 135 2 0 13 Aug 2024
Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts Weilin Cai Juyong Jiang Le Qin Junwei Cui Sunghun Kim Jiayi Huang 132 9 0 07 Apr 2024
Hash Layers For Large Sparse Models Stephen Roller Sainbayar Sukhbaatar Arthur Szlam Jason Weston MoE 181 214 0 08 Jun 2021
BASE Layers: Simplifying Training of Large, Sparse Models M. Lewis Shruti Bhosale Tim Dettmers Naman Goyal Luke Zettlemoyer MoE 201 281 0 30 Mar 2021
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity W. Fedus Barret Zoph Noam M. Shazeer MoE 88 2,226 0 11 Jan 2021
XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders Shuming Ma Jian Yang Haoyang Huang Zewen Chi Li Dong ... Akiko Eriguchi Saksham Singhal Xia Song Arul Menezes Furu Wei LRM 58 33 0 31 Dec 2020
Multi-task Learning for Multilingual Neural Machine Translation Yiren Wang Chengxiang Zhai Hany Awadalla 80 69 0 06 Oct 2020
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding Dmitry Lepikhin HyoukJoong Lee Yuanzhong Xu Dehao Chen Orhan Firat Yanping Huang M. Krikun Noam M. Shazeer Zhiwen Chen MoE 124 1,191 0 30 Jun 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 880 42,379 0 28 May 2020
UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training Hangbo Bao Li Dong Furu Wei Wenhui Wang Nan Yang ... Yu Wang Songhao Piao Jianfeng Gao Ming Zhou H. Hon AI4CE 88 394 0 28 Feb 2020
Unsupervised Cross-lingual Representation Learning at Scale Alexis Conneau Kartikay Khandelwal Naman Goyal Vishrav Chaudhary Guillaume Wenzek Francisco Guzmán Edouard Grave Myle Ott Luke Zettlemoyer Veselin Stoyanov 228 6,587 0 05 Nov 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 485 20,342 0 23 Oct 2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy M. Lewis Luke Zettlemoyer Veselin Stoyanov AIMat 686 24,557 0 26 Jul 2019
Unified Language Model Pre-training for Natural Language Understanding and Generation Li Dong Nan Yang Wenhui Wang Furu Wei Xiaodong Liu Yu Wang Jianfeng Gao M. Zhou H. Hon ELM AI4CE 227 1,560 0 08 May 2019
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 1.8K 95,229 0 11 Oct 2018
SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing Taku Kudo John Richardson 206 3,531 0 19 Aug 2018
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 789 132,454 0 12 Jun 2017
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer Noam M. Shazeer Azalia Mirhoseini Krzysztof Maziarz Andy Davis Quoc V. Le Geoffrey E. Hinton J. Dean MoE 253 2,686 0 23 Jan 2017
Neural Machine Translation of Rare Words with Subword Units Rico Sennrich Barry Haddow Alexandra Birch 232 7,760 0 31 Aug 2015
Adam: A Method for Stochastic Optimization Diederik P. Kingma Jimmy Ba ODL 2.1K 150,364 0 22 Dec 2014