Communication Efficient Distributed Training with Distributed Lion

Communication Efficient Distributed Training with Distributed Lion

30 March 2024

Raghuraman Krishnamoorthi

Papers citing "Communication Efficient Distributed Training with Distributed Lion"

12 / 12 papers shown

Title
Distributed Sign Momentum with Local Steps for Training Transformers Shuhua Yu Ding Zhou Cong Xie An Xu Zhi-Li Zhang Xin Liu S. Kar 94 0 0 26 Nov 2024
Cautious Optimizers: Improving Training with One Line of Code Kaizhao Liang Lizhang Chen B. Liu Qiang Liu ODL 152 8 0 25 Nov 2024
Asynchronous Local-SGD Training for Language Modeling Bo Liu Rachita Chhaparia Arthur Douillard Satyen Kale Andrei A. Rusu Jiajun Shen Arthur Szlam MarcÁurelio Ranzato FedML 55 11 0 17 Jan 2024
Symbolic Discovery of Optimization Algorithms Xiangning Chen Chen Liang Da Huang Esteban Real Kaiyuan Wang ... Xuanyi Dong Thang Luong Cho-Jui Hsieh Yifeng Lu Quoc V. Le 119 367 0 13 Feb 2023
Training Compute-Optimal Large Language Models Jordan Hoffmann Sebastian Borgeaud A. Mensch Elena Buchatskaya Trevor Cai ... Karen Simonyan Erich Elsen Jack W. Rae Oriol Vinyals Laurent Sifre AI4TS 139 1,915 0 29 Mar 2022
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions Christopher Clark Kenton Lee Ming-Wei Chang Tom Kwiatkowski Michael Collins Kristina Toutanova 191 1,475 0 24 May 2019
AutoAugment: Learning Augmentation Policies from Data E. D. Cubuk Barret Zoph Dandelion Mané Vijay Vasudevan Quoc V. Le 105 1,764 0 24 May 2018
Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training Chengyue Wu Song Han Huizi Mao Yu Wang W. Dally 107 1,399 0 05 Dec 2017
Asynchronous Stochastic Gradient Descent with Delay Compensation Shuxin Zheng Qi Meng Taifeng Wang Wei Chen Nenghai Yu Zhiming Ma Tie-Yan Liu 88 314 0 27 Sep 2016
Layer Normalization Jimmy Lei Ba J. Kiros Geoffrey E. Hinton 300 10,412 0 21 Jul 2016
Revisiting Distributed Synchronous SGD Jianmin Chen Xinghao Pan R. Monga Samy Bengio Rafal Jozefowicz 66 799 0 04 Apr 2016
ImageNet Large Scale Visual Recognition Challenge Olga Russakovsky Jia Deng Hao Su J. Krause S. Satheesh ... A. Karpathy A. Khosla Michael S. Bernstein Alexander C. Berg Li Fei-Fei VLM ObjD 1.2K 39,383 0 01 Sep 2014