GShard: Scaling Giant Models with Conditional Computation and Automatic
Sharding

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

30 June 2020

Dmitry Lepikhin

Noam M. Shazeer

Papers citing "GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding"

10 / 260 papers shown

Title
Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Data Jonathan Pilault Amine Elhattami C. Pal CLL MoE 24 89 0 19 Sep 2020
Efficient Transformers: A Survey Yi Tay Mostafa Dehghani Dara Bahri Donald Metzler VLM 114 1,102 0 14 Sep 2020
DAPPLE: A Pipelined Data Parallel Approach for Training Large Models Shiqing Fan Yi Rong Chen Meng Zongyan Cao Siyu Wang ... Jun Yang Lixue Xia Lansong Diao Xiaoyong Liu Wei Lin 21 232 0 02 Jul 2020
Enabling Compute-Communication Overlap in Distributed Deep Learning Training Platforms Saeed Rashidi Matthew Denton Srinivas Sridharan Sudarshan Srinivasan Amoghavarsha Suresh Jade Nie T. Krishna 26 45 0 30 Jun 2020
Meta Pseudo Labels Hieu H. Pham Zihang Dai Qizhe Xie Minh-Thang Luong Quoc V. Le VLM 262 656 0 23 Mar 2020
Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts Max Ryabinin Anton I. Gusev FedML 27 48 0 10 Feb 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 264 4,489 0 23 Jan 2020
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 245 1,826 0 17 Sep 2019
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation Yonghui Wu M. Schuster Z. Chen Quoc V. Le Mohammad Norouzi ... Alex Rudnick Oriol Vinyals G. Corrado Macduff Hughes J. Dean AIMat 716 6,746 0 26 Sep 2016
Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism Orhan Firat Kyunghyun Cho Yoshua Bengio LRM AIMat 231 623 0 06 Jan 2016