v1v2 (latest)

A Distributed Synchronous SGD Algorithm with Global Top- $k$ Sparsification for Low Bandwidth Networks

14 January 2019

Papers citing "A Distributed Synchronous SGD Algorithm with Global Top-$k$ Sparsification for Low Bandwidth Networks"

50 / 65 papers shown

Title
Exploring Scaling Laws for Local SGD in Large Language Model Training Qiaozhi He Xiaomin Zhuang Zhihua Wu 90 4 0 20 Sep 2024
Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning Zichen Tang Junlin Huang Rudan Yan Yuxin Wang Zhenheng Tang Shaoshuai Shi Amelie Chi Zhou Xiaowen Chu FedML 84 2 0 27 Aug 2024
Beyond Throughput and Compression Ratios: Towards High End-to-end Utility of Gradient Compression Wenchen Han S. Vargaftik Michael Mitzenmacher Brad Karp Ran Ben-Basat 80 3 0 01 Jul 2024
Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Survey Feng Liang Zhen Zhang Haifeng Lu Victor C. M. Leung Yanyi Guo Xiping Hu GNN 103 8 0 09 Apr 2024
Preserving Near-Optimal Gradient Sparsification Cost for Scalable Distributed Deep Learning Daegun Yoon Sangyoon Oh 69 0 0 21 Feb 2024
Accelerating Distributed Deep Learning using Lossless Homomorphic Compression Haoyu Li Yuchen Xu Jiayi Chen Rohit Dwivedula Wenfei Wu Keqiang He Aditya Akella Daehyeok Kim FedML AI4CE 42 4 0 12 Feb 2024
Improved Quantization Strategies for Managing Heavy-tailed Gradients in Distributed Learning Guangfeng Yan Tan Li Yuanzhang Xiao Hanxu Hou Linqi Song MQ 53 0 0 02 Feb 2024
Layered Randomized Quantization for Communication-Efficient and Privacy-Preserving Distributed Learning Guangfeng Yan Tan Li Tian-Shing Lan Kui Wu Linqi Song 44 7 0 12 Dec 2023
FedCode: Communication-Efficient Federated Learning via Transferring Codebooks Saeed Khalilian Gourtani Vasileios Tsouvalas T. Ozcelebi N. Meratnia FedML 100 5 0 15 Nov 2023
Near-Linear Scaling Data Parallel Training with Overlapping-Aware Gradient Compression Lin Meng Yuzhong Sun Weimin Li 77 1 0 08 Nov 2023
Inclusive Data Representation in Federated Learning: A Novel Approach Integrating Textual and Visual Prompt Zihao Zhao Zhenpeng Shi Yang Liu Wenbo Ding FedML 75 1 0 04 Oct 2023
MiCRO: Near-Zero Cost Gradient Sparsification for Scaling and Accelerating Distributed DNN Training Daegun Yoon Sangyoon Oh 64 1 0 02 Oct 2023
CORE: Common Random Reconstruction for Distributed Optimization with Provable Low Communication Complexity Pengyun Yue Hanzheng Zhao Cong Fang Di He Liwei Wang Zhouchen Lin Song-Chun Zhu 85 1 0 23 Sep 2023
FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs Zhenheng Tang Yuxin Wang Xin He Longteng Zhang Xinglin Pan ... Rongfei Zeng Kaiyong Zhao Shaoshuai Shi Bingsheng He Xiaowen Chu 106 30 0 03 Sep 2023
A Survey of What to Share in Federated Learning: Perspectives on Model Utility, Privacy Leakage, and Communication Efficiency Jiawei Shao Zijian Li Wenqiang Sun Tailin Zhou Yuchang Sun Lumin Liu Zehong Lin Yuyi Mao Jun Zhang FedML 109 28 0 20 Jul 2023
Accelerating Distributed ML Training via Selective Synchronization S. Tyagi Martin Swany FedML 98 4 0 16 Jul 2023
DEFT: Exploiting Gradient Norm Difference between Model Layers for Scalable Gradient Sparsification Daegun Yoon Sangyoon Oh 75 1 0 07 Jul 2023
Evaluation and Optimization of Gradient Compression for Distributed Deep Learning Lin Zhang Longteng Zhang Shaoshuai Shi Xiaowen Chu Yue Liu OffRL 45 7 0 15 Jun 2023
Killing Two Birds with One Stone: Quantization Achieves Privacy in Distributed Learning Guangfeng Yan Tan Li Kui Wu Linqi Song 58 13 0 26 Apr 2023
SparDL: Distributed Deep Learning Training with Efficient Sparse Communication Minjun Zhao Yichen Yin Yuren Mao Qing Liu Lu Chen Yunjun Gao 48 1 0 03 Apr 2023
AutoDDL: Automatic Distributed Deep Learning with Near-Optimal Bandwidth Cost Jinfan Chen Shigang Li Ran Guo Jinhui Yuan Torsten Hoefler 57 2 0 17 Jan 2023
Top-k data selection via distributed sample quantile inference Xu Zhang M. Vasconcelos 57 1 0 01 Dec 2022
Empirical Analysis on Top-k Gradient Sparsification for Distributed Deep Learning in a Supercomputing Environment Daegun Yoon Sangyoon Oh 85 0 0 18 Sep 2022
Towards Efficient Communications in Federated Learning: A Contemporary Survey Zihao Zhao Yuzhu Mao Yang Liu Linqi Song Ouyang Ye Xinlei Chen Wenbo Ding FedML 112 64 0 02 Aug 2022
Byzantine Fault Tolerance in Distributed Machine Learning : a Survey Djamila Bouhata Hamouma Moumen Moumen Hamouma Ahcène Bounceur AI4CE 106 8 0 05 May 2022
Near-Optimal Sparse Allreduce for Distributed Deep Learning Shigang Li Torsten Hoefler 62 53 0 19 Jan 2022
AET-SGD: Asynchronous Event-triggered Stochastic Gradient Descent Nhuong V. Nguyen Song Han 57 2 0 27 Dec 2021
Collaborative Learning over Wireless Networks: An Introductory Overview Emre Ozfatura Deniz Gunduz H. Vincent Poor 76 12 0 07 Dec 2021
FedSkel: Efficient Federated Learning on Heterogeneous Systems with Skeleton Gradients Update Junyu Luo Jianlei Yang Xucheng Ye Xin Guo Weisheng Zhao FedML 59 14 0 20 Aug 2021
A Distributed SGD Algorithm with Global Sketching for Deep Learning Training Acceleration Lingfei Dai Boyu Diao Chao Li Yongjun Xu 68 5 0 13 Aug 2021
BenchENAS: A Benchmarking Platform for Evolutionary Neural Architecture Search Xiangning Xie Yuqiao Liu Yanan Sun Gary G. Yen Bing Xue Mengjie Zhang 117 19 0 09 Aug 2021
ErrorCompensatedX: error compensation for variance reduced algorithms Hanlin Tang Yao Li Ji Liu Ming Yan 83 10 0 04 Aug 2021
Neural Distributed Source Coding Jay Whang Alliot Nagle Anish Acharya Hyeji Kim A. Dimakis 85 21 0 05 Jun 2021
ScaleCom: Scalable Sparsified Gradient Compression for Communication-Efficient Distributed Training Chia-Yu Chen Jiamin Ni Songtao Lu Xiaodong Cui Pin-Yu Chen ... Naigang Wang Swagath Venkataramani Vijayalakshmi Srinivasan Wei Zhang K. Gopalakrishnan 79 67 0 21 Apr 2021
1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed Conglong Li A. A. Awan Hanlin Tang Samyam Rajbhandari Yuxiong He 115 33 0 13 Apr 2021
On the Utility of Gradient Compression in Distributed Training Systems Saurabh Agarwal Hongyi Wang Shivaram Venkataraman Dimitris Papailiopoulos 107 47 0 28 Feb 2021
Peering Beyond the Gradient Veil with Distributed Auto Differentiation Bradley T. Baker Aashis Khanal Vince D. Calhoun Barak A. Pearlmutter Sergey Plis 47 1 0 18 Feb 2021
Federated Learning over Wireless Networks: A Band-limited Coordinated Descent Approach Junshan Zhang Na Li M. Dedeoglu FedML 72 41 0 16 Feb 2021
Distributed Online Learning for Joint Regret with Communication Constraints Dirk van der Hoeven Hédi Hadiji T. Erven 64 5 0 15 Feb 2021
1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed Hanlin Tang Shaoduo Gan A. A. Awan Samyam Rajbhandari Conglong Li Xiangru Lian Ji Liu Ce Zhang Yuxiong He AI4CE 102 87 0 04 Feb 2021
FEDZIP: A Compression Framework for Communication-Efficient Federated Learning Amirhossein Malekijoo Mohammad Javad Fadaeieslam Hanieh Malekijou Morteza Homayounfar F. Alizadeh-Shabdiz Reza Rawassizadeh FedML 94 55 0 02 Feb 2021
Time-Correlated Sparsification for Communication-Efficient Federated Learning Emre Ozfatura Kerem Ozfatura Deniz Gunduz FedML 89 49 0 21 Jan 2021
Bayesian Federated Learning over Wireless Networks Seunghoon Lee Chanhoo Park Songnam Hong Yonina C. Eldar Namyoon Lee 70 23 0 31 Dec 2020
Distributed Sparse SGD with Majority Voting Kerem Ozfatura Emre Ozfatura Deniz Gunduz FedML 76 4 0 12 Nov 2020
Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification Saurabh Agarwal Hongyi Wang Kangwook Lee Shivaram Venkataraman Dimitris Papailiopoulos 85 25 0 29 Oct 2020
Hogwild! over Distributed Local Data Sets with Linearly Increasing Mini-Batch Sizes Marten van Dijk Nhuong V. Nguyen Toan N. Nguyen Lam M. Nguyen Quoc Tran-Dinh Phuong Ha Nguyen FedML 116 10 0 27 Oct 2020
Towards Scalable Distributed Training of Deep Learning on Public Cloud Clusters Shaoshuai Shi Xianhao Zhou Shutao Song Xingyao Wang Zilin Zhu ... Chenyang Guo Bo Yang Zhibo Chen Yongjian Wu Xiaowen Chu GNN 81 55 0 20 Oct 2020
PSO-PS: Parameter Synchronization with Particle Swarm Optimization for Distributed Training of Deep Neural Networks Qing Ye Y. Han Yanan Sun Jiancheng Lv 47 3 0 06 Sep 2020
APMSqueeze: A Communication Efficient Adam-Preconditioned Momentum SGD Algorithm Hanlin Tang Shaoduo Gan Samyam Rajbhandari Xiangru Lian Ji Liu Yuxiong He Ce Zhang 82 8 0 26 Aug 2020
DBS: Dynamic Batch Size For Distributed Deep Neural Network Training Qing Ye Yuhao Zhou Mingjia Shi Yanan Sun Jiancheng Lv 57 11 0 23 Jul 2020

A Distributed Synchronous SGD Algorithm with Global Top-kkk Sparsification for Low Bandwidth Networks

Papers citing "A Distributed Synchronous SGD Algorithm with Global Top-$k$ Sparsification for Low Bandwidth Networks"

A Distributed Synchronous SGD Algorithm with Global Top- $k$ Sparsification for Low Bandwidth Networks