v1v2v3 (latest)

Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts

7 April 2024

Papers citing "Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts"

24 / 74 papers shown

Title
Rethinking Attention with Performers K. Choromanski Valerii Likhosherstov David Dohan Xingyou Song Andreea Gane ... Afroz Mohiuddin Lukasz Kaiser David Belanger Lucy J. Colwell Adrian Weller 186 1,602 0 30 Sep 2020
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding Dmitry Lepikhin HyoukJoong Lee Yuanzhong Xu Dehao Chen Orhan Firat Yanping Huang M. Krikun Noam M. Shazeer Zhiwen Chen MoE 124 1,191 0 30 Jun 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 882 42,463 0 28 May 2020
On Layer Normalization in the Transformer Architecture Ruibin Xiong Yunchang Yang Di He Kai Zheng Shuxin Zheng Chen Xing Huishuai Zhang Yanyan Lan Liwei Wang Tie-Yan Liu AI4CE 145 996 0 12 Feb 2020
Reformer: The Efficient Transformer Nikita Kitaev Lukasz Kaiser Anselm Levskaya VLM 204 2,333 0 13 Jan 2020
PIQA: Reasoning about Physical Commonsense in Natural Language Yonatan Bisk Rowan Zellers Ronan Le Bras Jianfeng Gao Yejin Choi OOD LRM 178 1,847 0 26 Nov 2019
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models Samyam Rajbhandari Jeff Rasley Olatunji Ruwase Yuxiong He ALM AI4CE 82 919 0 04 Oct 2019
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations Zhenzhong Lan Mingda Chen Sebastian Goodman Kevin Gimpel Piyush Sharma Radu Soricut SSL AIMat 373 6,469 0 26 Sep 2019
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks Jiasen Lu Dhruv Batra Devi Parikh Stefan Lee SSL VLM 249 3,695 0 06 Aug 2019
MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms Aida Amini Saadia Gabriel Shanchuan Lin Rik Koncel-Kedziorski Yejin Choi Hannaneh Hajishirzi AIMat ReLM AI4CE 127 581 0 30 May 2019
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions Christopher Clark Kenton Lee Ming-Wei Chang Tom Kwiatkowski Michael Collins Kristina Toutanova 244 1,560 0 24 May 2019
HellaSwag: Can a Machine Really Finish Your Sentence? Rowan Zellers Ari Holtzman Yonatan Bisk Ali Farhadi Yejin Choi 182 2,532 0 19 May 2019
fairseq: A Fast, Extensible Toolkit for Sequence Modeling Myle Ott Sergey Edunov Alexei Baevski Angela Fan Sam Gross Nathan Ng David Grangier Michael Auli VLM FaML 120 3,156 0 01 Apr 2019
Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools R. Mayer Hans-Arno Jacobsen GNN 74 190 0 27 Mar 2019
Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect Ang Li Shuaiwen Leon Song Jieyang Chen Jiajia Li Xu Liu Nathan R. Tallent Kevin J. Barker GNN 90 219 0 11 Mar 2019
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering Todor Mihaylov Peter Clark Tushar Khot Ashish Sabharwal 119 1,570 0 08 Sep 2018
Universal Transformers Mostafa Dehghani Stephan Gouws Oriol Vinyals Jakob Uszkoreit Lukasz Kaiser 87 755 0 10 Jul 2018
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge Peter Clark Isaac Cowhey Oren Etzioni Tushar Khot Ashish Sabharwal Carissa Schoenick Oyvind Tafjord ELM RALM LRM 172 2,670 0 14 Mar 2018
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 798 132,454 0 12 Jun 2017
RACE: Large-scale ReAding Comprehension Dataset From Examinations Guokun Lai Qizhe Xie Hanxiao Liu Yiming Yang Eduard H. Hovy ELM 195 1,359 0 15 Apr 2017
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer Noam M. Shazeer Azalia Mirhoseini Krzysztof Maziarz Andy Davis Quoc V. Le Geoffrey E. Hinton J. Dean MoE 253 2,692 0 23 Jan 2017
Pointer Sentinel Mixture Models Stephen Merity Caiming Xiong James Bradbury R. Socher RALM 343 2,900 0 26 Sep 2016
Densely Connected Convolutional Networks Gao Huang Zhuang Liu Laurens van der Maaten Kilian Q. Weinberger PINN 3DV 802 36,892 0 25 Aug 2016
Distributed Representations of Words and Phrases and their Compositionality Tomas Mikolov Ilya Sutskever Kai Chen G. Corrado J. Dean NAI OCL 402 33,565 0 16 Oct 2013