Layerwise Recurrent Router for Mixture-of-Experts

Layerwise Recurrent Router for Mixture-of-Experts

13 August 2024

Zili Wang

Papers citing "Layerwise Recurrent Router for Mixture-of-Experts"

8 / 8 papers shown

Title
Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models Z. Qiu Zeyu Huang Bo Zheng Kaiyue Wen Z. Wang Rui Men Ivan Titov Dayiheng Liu Jingren Zhou Junyang Lin MoE 57 6 0 21 Jan 2025
Post-hoc Reward Calibration: A Case Study on Length Bias Zeyu Huang Zihan Qiu Zili Wang Edoardo M. Ponti Ivan Titov 40 5 0 25 Sep 2024
Harder Tasks Need More Experts: Dynamic Routing in MoE Models Quzhe Huang Zhenwei An Zhuang Nan Mingxu Tao Chen Zhang ... Kun Xu Kun Xu Liwei Chen Songfang Huang Yansong Feng MoE 37 26 0 12 Mar 2024
DeepSeek LLM: Scaling Open-Source Language Models with Longtermism DeepSeek-AI Xiao Bi : Xiao Bi Deli Chen Guanting Chen ... Yao Zhao Shangyan Zhou Shunfeng Zhou Qihao Zhu Yuheng Zou LRM ALM 139 306 0 05 Jan 2024
Mixture of Attention Heads: Selecting Attention Heads Per Token Xiaofeng Zhang Yikang Shen Zeyu Huang Jie Zhou Wenge Rong Zhang Xiong MoE 99 42 0 11 Oct 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 311 11,915 0 04 Mar 2022
Mixture-of-Experts with Expert Choice Routing Yan-Quan Zhou Tao Lei Han-Chu Liu Nan Du Yanping Huang Vincent Zhao Andrew M. Dai Zhifeng Chen Quoc V. Le James Laudon MoE 160 327 0 18 Feb 2022
Neural Architecture Search with Reinforcement Learning Barret Zoph Quoc V. Le 264 5,326 0 05 Nov 2016