On the Utility of Gradient Compression in Distributed Training Systems

28 February 2021

Saurabh Agarwal

Hongyi Wang

Shivaram Venkataraman

Dimitris Papailiopoulos

ArXiv PDF HTML

Papers citing "On the Utility of Gradient Compression in Distributed Training Systems"

27 / 27 papers shown

Title
OmniLearn: A Framework for Distributed Deep Learning over Heterogeneous Clusters S. Tyagi Prateek Sharma 63 0 0 21 Mar 2025
Beyond Throughput and Compression Ratios: Towards High End-to-end Utility of Gradient Compression Wenchen Han S. Vargaftik Michael Mitzenmacher Brad Karp Ran Ben-Basat 35 2 0 01 Jul 2024
Approximated Likelihood Ratio: A Forward-Only and Parallel Framework for Boosting Neural Network Training Zeliang Zhang Jinyang Jiang Zhuo Liu Susan Liang Yijie Peng Chenliang Xu 29 0 0 18 Mar 2024
Breaking MLPerf Training: A Case Study on Optimizing BERT Yongdeok Kim Jaehyung Ahn Myeongwoo Kim Changin Choi Heejae Kim ... Xiongzhan Linghu Jingkun Ma Lin Chen Yuehua Dai Sungjoo Yoo 25 0 0 04 Feb 2024
Flexible Communication for Optimal Distributed Learning over Unpredictable Networks S. Tyagi Martin Swany 37 1 0 05 Dec 2023
Near-Linear Scaling Data Parallel Training with Overlapping-Aware Gradient Compression Lin Meng Yuzhong Sun Weimin Li 34 1 0 08 Nov 2023
CacheGen: KV Cache Compression and Streaming for Fast Language Model Serving Yuhan Liu Hanchen Li Yihua Cheng Siddhant Ray Yuyang Huang ... Ganesh Ananthanarayanan Michael Maire Henry Hoffmann Ari Holtzman Junchen Jiang 50 41 0 11 Oct 2023
Accelerating Distributed ML Training via Selective Synchronization S. Tyagi Martin Swany FedML 29 3 0 16 Jul 2023
Just One Byte (per gradient): A Note on Low-Bandwidth Decentralized Language Model Finetuning Using Shared Randomness E. Zelikman Qian Huang Percy Liang Nick Haber Noah D. Goodman 62 14 0 16 Jun 2023
Evaluation and Optimization of Gradient Compression for Distributed Deep Learning Lin Zhang Longteng Zhang S. Shi X. Chu Bo-wen Li OffRL 23 7 0 15 Jun 2023
Adaptive Message Quantization and Parallelization for Distributed Full-graph GNN Training Borui Wan Juntao Zhao Chuan Wu GNN 14 14 0 02 Jun 2023
Global-QSGD: Practical Floatless Quantization for Distributed Learning with Theoretical Guarantees Jihao Xin Marco Canini Peter Richtárik Samuel Horváth 25 2 0 29 May 2023
GraVAC: Adaptive Compression for Communication-Efficient Distributed DL Training S. Tyagi Martin Swany 25 4 0 20 May 2023
THC: Accelerating Distributed Deep Learning Using Tensor Homomorphic Compression Minghao Li Ran Ben-Basat S. Vargaftik Chon-In Lao Ke Xu Michael Mitzenmacher Minlan Yu Harvard University 24 15 0 16 Feb 2023
Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression Jaeyong Song Jinkyu Yim Jaewon Jung Hongsun Jang H. Kim Youngsok Kim Jinho Lee GNN 16 25 0 24 Jan 2023
Does compressing activations help model parallel training? S. Bian Dacheng Li Hongyi Wang Eric P. Xing Shivaram Venkataraman 19 5 0 06 Jan 2023
Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities Brian Bartoldson B. Kailkhura Davis W. Blalock 31 47 0 13 Oct 2022
ByteComp: Revisiting Gradient Compression in Distributed Training Zhuang Wang Haibin Lin Yibo Zhu T. Ng 11 2 0 28 May 2022
Distributed Learning With Sparsified Gradient Differences Yicheng Chen Rick S. Blum Martin Takáč Brian M. Sadler 21 15 0 05 Feb 2022
FedLite: A Scalable Approach for Federated Learning on Resource-constrained Clients Jianyu Wang Qi A. S. Rawat Sashank J. Reddi Sagar M. Waghmare Felix X. Yu Gauri Joshi FedML 22 22 0 28 Jan 2022
CGX: Adaptive System Support for Communication-Efficient Deep Learning I. Markov Hamidreza Ramezanikebrya Dan Alistarh GNN 17 3 0 16 Nov 2021
A Field Guide to Federated Optimization Jianyu Wang Zachary B. Charles Zheng Xu Gauri Joshi H. B. McMahan ... Mi Zhang Tong Zhang Chunxiang Zheng Chen Zhu Wennan Zhu FedML 184 411 0 14 Jul 2021
JUWELS Booster -- A Supercomputer for Large-Scale AI Research Stefan Kesselheim A. Herten K. Krajsek J. Ebert J. Jitsev ... A. Strube Roshni Kamath Martin G. Schultz M. Riedel T. Lippert GNN 23 14 0 30 Jun 2021
Zero-Shot Text-to-Image Generation Aditya A. Ramesh Mikhail Pavlov Gabriel Goh Scott Gray Chelsea Voss Alec Radford Mark Chen Ilya Sutskever VLM 255 4,777 0 24 Feb 2021
IntSGD: Adaptive Floatless Compression of Stochastic Gradients Konstantin Mishchenko Bokun Wang D. Kovalev Peter Richtárik 67 14 0 16 Feb 2021
An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systems A. Abdelmoniem Ahmed Elzanaty Mohamed-Slim Alouini Marco Canini 51 74 0 26 Jan 2021
Communication-Efficient Distributed Deep Learning: A Comprehensive Survey Zhenheng Tang S. Shi Wei Wang Bo-wen Li Xiaowen Chu 19 48 0 10 Mar 2020