AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training

7 December 2017

Papers citing "AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training"

50 / 64 papers shown

Title
Novel Gradient Sparsification Algorithm via Bayesian Inference Ali Bereyhi B. Liang G. Boudreau Ali Afana 61 2 0 23 Sep 2024
I/O in Machine Learning Applications on HPC Systems: A 360-degree Survey Noah Lewis J. L. Bez Suren Byna 109 0 0 16 Apr 2024
Smart-Infinity: Fast Large Language Model Training using Near-Storage Processing on a Real System Hongsun Jang Jaeyong Song Jaewon Jung Jaeyoung Park Youngsok Kim Jinho Lee 41 16 0 11 Mar 2024
Preserving Near-Optimal Gradient Sparsification Cost for Scalable Distributed Deep Learning Daegun Yoon Sangyoon Oh 69 0 0 21 Feb 2024
Communication-Efficient Distributed Learning with Local Immediate Error Compensation Yifei Cheng Li Shen Linli Xu Xun Qian Shiwei Wu Yiming Zhou Tie Zhang Dacheng Tao Enhong Chen 65 0 0 19 Feb 2024
Temporal Knowledge Distillation for Time-Sensitive Financial Services Applications Hongda Shen Eren Kurshan AAML 49 2 0 28 Dec 2023
FedSZ: Leveraging Error-Bounded Lossy Compression for Federated Learning Communications Grant Wilkins Sheng Di Jon C. Calhoun Zilinghan Li Kibaek Kim Robert Underwood Richard Mortier Franck Cappello FedML 78 4 0 20 Dec 2023
Near-Linear Scaling Data Parallel Training with Overlapping-Aware Gradient Compression Lin Meng Yuzhong Sun Weimin Li 77 1 0 08 Nov 2023
MiCRO: Near-Zero Cost Gradient Sparsification for Scaling and Accelerating Distributed DNN Training Daegun Yoon Sangyoon Oh 64 1 0 02 Oct 2023
DEFT: Exploiting Gradient Norm Difference between Model Layers for Scalable Gradient Sparsification Daegun Yoon Sangyoon Oh 71 1 0 07 Jul 2023
Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression Jaeyong Song Jinkyu Yim Jaewon Jung Hongsun Jang H. Kim Youngsok Kim Jinho Lee GNN 67 28 0 24 Jan 2023
L-GreCo: Layerwise-Adaptive Gradient Compression for Efficient and Accurate Deep Learning Mohammadreza Alimohammadi I. Markov Elias Frantar Dan Alistarh 73 5 0 31 Oct 2022
Approximate Computing and the Efficient Machine Learning Expedition J. Henkel Hai Helen Li A. Raghunathan M. Tahoori Swagath Venkataramani Xiaoxuan Yang Georgios Zervakis 49 17 0 02 Oct 2022
Empirical Analysis on Top-k Gradient Sparsification for Distributed Deep Learning in a Supercomputing Environment Daegun Yoon Sangyoon Oh 80 0 0 18 Sep 2022
Reconciling Security and Communication Efficiency in Federated Learning Karthik Prasad Sayan Ghosh Graham Cormode Ilya Mironov Ashkan Yousefpour Pierre Stock FedML 69 9 0 26 Jul 2022
sqSGD: Locally Private and Communication Efficient Federated Learning Yan Feng Tao Xiong Ruofan Wu Lingjuan Lv Leilei Shi FedML 70 2 0 21 Jun 2022
Rate-Distortion Theoretic Bounds on Generalization Error for Distributed Learning Romain Chor Abdellatif Zaidi Milad Sefidgaran FedML 81 15 0 06 Jun 2022
ByteComp: Revisiting Gradient Compression in Distributed Training Zhuang Wang Yanghua Peng Yibo Zhu T. Ng 59 2 0 28 May 2022
Efficient Direct-Connect Topologies for Collective Communications Liangyu Zhao Siddharth Pal Tapan Chugh Weiyang Wang Jason Fantl P. Basu J. Khoury Arvind Krishnamurthy 86 7 0 07 Feb 2022
TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed Training Jobs Weiyang Wang Moein Khazraee Zhizhen Zhong M. Ghobadi Zhihao Jia Dheevatsa Mudigere Ying Zhang A. Kewitsch 123 93 0 01 Feb 2022
ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction Keshav Santhanam Omar Khattab Jon Saad-Falcon Christopher Potts Matei A. Zaharia 128 417 0 02 Dec 2021
Doing More by Doing Less: How Structured Partial Backpropagation Improves Deep Learning Clusters Adarsh Kumar Kausik Subramanian Shivaram Venkataraman Aditya Akella 30 5 0 20 Nov 2021
CGX: Adaptive System Support for Communication-Efficient Deep Learning I. Markov Hamidreza Ramezanikebrya Dan Alistarh GNN 82 5 0 16 Nov 2021
Resource-Efficient Federated Learning A. Abdelmoniem Atal Narayan Sahu Marco Canini Suhaib A. Fahmy FedML 91 57 0 01 Nov 2021
Large-Scale Deep Learning Optimizations: A Comprehensive Survey Xiaoxin He Fuzhao Xue Xiaozhe Ren Yang You 83 15 0 01 Nov 2021
Revealing and Protecting Labels in Distributed Training Trung D. Q. Dang Om Thakkar Swaroop Indra Ramaswamy Rajiv Mathews Peter Chin Franccoise Beaufays 38 26 0 31 Oct 2021
A Distributed SGD Algorithm with Global Sketching for Deep Learning Training Acceleration Lingfei Dai Boyu Diao Chao Li Yongjun Xu 63 5 0 13 Aug 2021
CD-SGD: Distributed Stochastic Gradient Descent with Compression and Delay Compensation Enda Yu Dezun Dong Yemao Xu Shuo Ouyang Xiangke Liao 47 5 0 21 Jun 2021
ScaleCom: Scalable Sparsified Gradient Compression for Communication-Efficient Distributed Training Chia-Yu Chen Jiamin Ni Songtao Lu Xiaodong Cui Pin-Yu Chen ... Naigang Wang Swagath Venkataramani Vijayalakshmi Srinivasan Wei Zhang K. Gopalakrishnan 79 67 0 21 Apr 2021
MergeComp: A Compression Scheduler for Scalable Communication-Efficient Distributed Training Zhuang Wang X. Wu T. Ng GNN 24 4 0 28 Mar 2021
Pufferfish: Communication-efficient Models At No Extra Cost Hongyi Wang Saurabh Agarwal Dimitris Papailiopoulos 85 59 0 05 Mar 2021
On the Impact of Device and Behavioral Heterogeneity in Federated Learning A. Abdelmoniem Chen-Yu Ho Pantelis Papageorgiou Muhammad Bilal Marco Canini FedML 59 18 0 15 Feb 2021
An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systems A. Abdelmoniem Ahmed Elzanaty Mohamed-Slim Alouini Marco Canini 123 77 0 26 Jan 2021
DynaComm: Accelerating Distributed CNN Training between Edges and Clouds through Dynamic Communication Scheduling Shangming Cai Dongsheng Wang Haixia Wang Yongqiang Lyu Guangquan Xu Xi Zheng A. Vasilakos 59 6 0 20 Jan 2021
Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification Saurabh Agarwal Hongyi Wang Kangwook Lee Shivaram Venkataraman Dimitris Papailiopoulos 85 25 0 29 Oct 2020
Fairness-aware Agnostic Federated Learning Wei Du Depeng Xu Xintao Wu Hanghang Tong FedML 90 131 0 10 Oct 2020
Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers Robin M. Schmidt Frank Schneider Philipp Hennig ODL 213 168 0 03 Jul 2020
Is Network the Bottleneck of Distributed Training? Zhen Zhang Chaokun Chang Yanghua Peng Yida Wang R. Arora Xin Jin 89 71 0 17 Jun 2020
Characterizing Impacts of Heterogeneity in Federated Learning upon Large-Scale Smartphone Data Chengxu Yang Qipeng Wang Mengwei Xu Shangguang Wang Kaigui Bian Yunxin Liu Xuanzhe Liu 100 23 0 12 Jun 2020
Communication-Efficient Distributed Deep Learning: A Comprehensive Survey Zhenheng Tang Shaoshuai Shi Wei Wang Yue Liu Xiaowen Chu 80 49 0 10 Mar 2020
Communication optimization strategies for distributed deep neural network training: A survey Shuo Ouyang Dezun Dong Yemao Xu Liquan Xiao 116 12 0 06 Mar 2020
Communication-Efficient Decentralized Learning with Sparsification and Adaptive Peer Selection Zhenheng Tang Shaoshuai Shi Xiaowen Chu FedML 62 58 0 22 Feb 2020
Communication-Efficient Edge AI: Algorithms and Systems Yuanming Shi Kai Yang Tao Jiang Jun Zhang Khaled B. Letaief GNN 99 334 0 22 Feb 2020
Intermittent Pulling with Local Compensation for Communication-Efficient Federated Learning Yining Qi Zhihao Qu Song Guo Xin Gao Ruixuan Li Baoliu Ye FedML 38 9 0 22 Jan 2020
Adaptive Gradient Sparsification for Efficient Federated Learning: An Online Learning Approach Pengchao Han Shiqiang Wang K. Leung FedML 78 182 0 14 Jan 2020
Understanding Top-k Sparsification in Distributed Deep Learning Shaoshuai Shi Xiaowen Chu Ka Chun Cheung Simon See 233 101 0 20 Nov 2019
Layer-wise Adaptive Gradient Sparsification for Distributed Deep Learning with Convergence Guarantees Shaoshuai Shi Zhenheng Tang Qiang-qiang Wang Kaiyong Zhao Xiaowen Chu 65 22 0 20 Nov 2019
On the Discrepancy between the Theoretical Analysis and Practical Implementations of Compressed Communication for Distributed Deep Learning Aritra Dutta El Houcine Bergou A. Abdelmoniem Chen-Yu Ho Atal Narayan Sahu Marco Canini Panos Kalnis 77 78 0 19 Nov 2019
Hyper-Sphere Quantization: Communication-Efficient SGD for Federated Learning XINYAN DAI Xiao Yan Kaiwen Zhou Han Yang K. K. Ng James Cheng Yu Fan FedML 68 47 0 12 Nov 2019
Gradient Sparification for Asynchronous Distributed Training Zijie Yan FedML 21 1 0 24 Oct 2019