TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning

22 May 2017

Yiran Chen

Papers citing "TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning"

50 / 467 papers shown

Title
Striving for Simplicity: Simple Yet Effective Prior-Aware Pseudo-Labeling for Semi-Supervised Ultrasound Image Segmentation Yaxiong Chen Yujie Wang Zixuan Zheng Jingliang Hu Yilei Shi Shengwu Xiong Xiao Xiang Zhu Lichao Mou 59 0 0 18 Mar 2025
Layer-wise Update Aggregation with Recycling for Communication-Efficient Federated Learning Jisoo Kim Sungmin Kang Sunwoo Lee FedML 52 0 0 14 Mar 2025
Accelerated Distributed Optimization with Compression and Error Feedback Yuan Gao Anton Rodomanov Jeremy Rack Sebastian U. Stich 59 0 0 11 Mar 2025
MARINA-P: Superior Performance in Non-smooth Federated Optimization with Adaptive Stepsizes Igor Sokolov Peter Richtárik 90 1 0 22 Dec 2024
Lion Cub: Minimizing Communication Overhead in Distributed Lion Satoki Ishikawa Tal Ben-Nun B. Van Essen Rio Yokota Nikoli Dryden 84 0 0 25 Nov 2024
Problem-dependent convergence bounds for randomized linear gradient compression Thomas Flynn Patrick R. Johnstone Shinjae Yoo 81 0 0 19 Nov 2024
$r$ Age- $k$ : Communication-Efficient Federated Learning Using Age Factor Matin Mortaheb Priyanka Kaswan S. Ulukus FedML 36 1 0 29 Oct 2024
Age-of-Gradient Updates for Federated Learning over Random Access Channels Yu Heng Wu Houman Asgari Stefano Rini Andrea Munari FedML 26 1 0 15 Oct 2024
Asynchronous Stochastic Gradient Descent with Decoupled Backpropagation and Layer-Wise Updates Cabrel Teguemne Fokam Khaleelulla Khan Nazeer Lukas König David Kappel Anand Subramoney 37 0 0 08 Oct 2024
Memory-Efficient LLM Training with Online Subspace Descent Kaizhao Liang Bo Liu Lizhang Chen Qiang Liu 29 7 0 23 Aug 2024
A Joint Approach to Local Updating and Gradient Compression for Efficient Asynchronous Federated Learning Jiajun Song Jiajun Luo Rongwei Lu Shuzhao Xie Bin Chen Zhi Wang FedML 33 0 0 06 Jul 2024
LoCo: Low-Bit Communication Adaptor for Large-scale Model Training Xingyu Xie Zhijie Lin Kim-Chuan Toh Pan Zhou 42 2 0 05 Jul 2024
Beyond Throughput and Compression Ratios: Towards High End-to-end Utility of Gradient Compression Wenchen Han S. Vargaftik Michael Mitzenmacher Brad Karp Ran Ben-Basat 45 2 0 01 Jul 2024
FedEx: Expediting Federated Learning over Heterogeneous Mobile Devices by Overlapping and Participant Selection Jiaxiang Geng Boyu Li Xiaoqi Qin Yixuan Li Liang Li Yanzhao Hou Miao Pan FedML 42 0 0 01 Jul 2024
FedAQ: Communication-Efficient Federated Edge Learning via Joint Uplink and Downlink Adaptive Quantization Linping Qu Shenghui Song Chi-Ying Tsui MQ FedML 26 4 0 26 Jun 2024
Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradients Aashiq Muhamed Oscar Li David Woodruff Mona Diab Virginia Smith 61 8 0 25 Jun 2024
FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusion Li-Wen Chang Yiyuan Ma Qi Hou Chengquan Jiang Ningxin Zheng ... Zuquan Song Ziheng Jiang Yanghua Peng Xuanzhe Liu Xin Liu 41 23 0 11 Jun 2024
VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projections Roy Miles Pradyumna Reddy Ismail Elezi Jiankang Deng VLM 43 3 0 28 May 2024
Full-Stack Allreduce on Multi-Rail Networks Enda Yu Dezun Dong Xiangke Liao GNN 32 0 0 28 May 2024
What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions Sang Keun Choe Hwijeen Ahn Juhan Bae Kewen Zhao Minsoo Kang ... Teruko Mitamura Jeff Schneider Eduard Hovy Roger C. Grosse Eric Xing TDI 47 28 0 22 May 2024
Improved Communication-Privacy Trade-offs in $L_2$ Mean Estimation under Streaming Differential Privacy Wei-Ning Chen Berivan Isik Peter Kairouz Albert No Sewoong Oh Zheng Xu 60 3 0 02 May 2024
Approximate Wireless Communication for Lossy Gradient Updates in IoT Federated Learning Xiang Ma Haijian Sun Rose Qingyang Hu Yi Qian 58 0 0 17 Apr 2024
Hybrid FedGraph: An efficient hybrid federated learning algorithm using graph convolutional neural network Jaeyeon Jang Diego Klabjan Veena Mendiratta Fanfei Meng FedML 38 1 0 15 Apr 2024
Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Survey Feng Liang Zhen Zhang Haifeng Lu Victor C. M. Leung Yanyi Guo Xiping Hu GNN 39 6 0 09 Apr 2024
A Survey on Error-Bounded Lossy Compression for Scientific Datasets Sheng Di Jinyang Liu Kai Zhao Xin Liang Robert Underwood ... Jon C. Calhoun Guanpeng Li Kazutomo Yoshii Khalid Ayed Alharthi Franck Cappello AI4CE 39 14 0 03 Apr 2024
Compressed Federated Reinforcement Learning with a Generative Model Ali Beikmohammadi Sarit Khirirat Sindri Magnússon FedML 41 2 0 26 Mar 2024
SignSGD with Federated Voting Chanho Park H. Vincent Poor Namyoon Lee FedML 42 1 0 25 Mar 2024
Quantization Avoids Saddle Points in Distributed Optimization Yanan Bo Yongqiang Wang MQ 29 2 0 15 Mar 2024
Random Aggregate Beamforming for Over-the-Air Federated Learning in Large-Scale Networks Chunmei Xu Shengheng Liu Yongming Huang Björn E. Ottersten Dusist Niyato FedML 31 2 0 20 Feb 2024
Communication-Efficient Distributed Learning with Local Immediate Error Compensation Yifei Cheng Li Shen Linli Xu Xun Qian Shiwei Wu Yiming Zhou Tie Zhang Dacheng Tao Enhong Chen 45 0 0 19 Feb 2024
Smart Information Exchange for Unsupervised Federated Learning via Reinforcement Learning Seohyun Lee Anindya Bijoy Das Satyavrat Wagle Christopher G. Brinton FedML 30 0 0 15 Feb 2024
Accelerating Distributed Deep Learning using Lossless Homomorphic Compression Haoyu Li Yuchen Xu Jiayi Chen Rohit Dwivedula Wenfei Wu Keqiang He Aditya Akella Daehyeok Kim FedML AI4CE 29 4 0 12 Feb 2024
Expediting In-Network Federated Learning by Voting-Based Consensus Model Compression Xiaoxin Su Yipeng Zhou Laizhong Cui Song Guo FedML 30 3 0 06 Feb 2024
Fed-CVLC: Compressing Federated Learning Communications with Variable-Length Codes Xiaoxin Su Yipeng Zhou Laizhong Cui John C. S. Lui Jiangchuan Liu FedML 44 1 0 06 Feb 2024
Breaking MLPerf Training: A Case Study on Optimizing BERT Yongdeok Kim Jaehyung Ahn Myeongwoo Kim Changin Choi Heejae Kim ... Xiongzhan Linghu Jingkun Ma Lin Chen Yuehua Dai Sungjoo Yoo 32 0 0 04 Feb 2024
Improved Quantization Strategies for Managing Heavy-tailed Gradients in Distributed Learning Guangfeng Yan Tan Li Yuanzhang Xiao Hanxu Hou Linqi Song MQ 29 0 0 02 Feb 2024
Truncated Non-Uniform Quantization for Distributed SGD Guangfeng Yan Tan Li Yuanzhang Xiao Congduan Li Linqi Song MQ 22 0 0 02 Feb 2024
RS-DGC: Exploring Neighborhood Statistics for Dynamic Gradient Compression on Remote Sensing Image Interpretation Weiying Xie Zixuan Wang Jitao Ma Daixun Li Yunsong Li 38 0 0 29 Dec 2023
FedSZ: Leveraging Error-Bounded Lossy Compression for Federated Learning Communications Grant Wilkins Sheng Di Jon C. Calhoun Zilinghan Li Kibaek Kim Robert Underwood Richard Mortier Franck Cappello FedML 45 2 0 20 Dec 2023
Fed-QSSL: A Framework for Personalized Federated Learning under Bitwidth and Data Heterogeneity Yiyue Chen H. Vikalo C. Wang FedML 49 5 0 20 Dec 2023
Contractive error feedback for gradient compression Bingcong Li Shuai Zheng Parameswaran Raman Anshumali Shrivastava G. Giannakis 23 0 0 13 Dec 2023
Kimad: Adaptive Gradient Compression with Bandwidth Awareness Jihao Xin Ivan Ilin Shunkang Zhang Marco Canini Peter Richtárik 45 3 0 13 Dec 2023
The Landscape of Modern Machine Learning: A Review of Machine, Distributed and Federated Learning Omer Subasi Oceane Bel Joseph Manzano Kevin J. Barker FedML OOD PINN 30 2 0 05 Dec 2023
Flexible Communication for Optimal Distributed Learning over Unpredictable Networks S. Tyagi Martin Swany 47 1 0 05 Dec 2023
Edge AI for Internet of Energy: Challenges and Perspectives Yassine Himeur A. Sayed A. Alsalemi F. Bensaali Abbes Amira 59 26 0 28 Nov 2023
ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization Prateek Yadav Leshem Choshen Colin Raffel Mohit Bansal 34 13 0 22 Nov 2023
EControl: Fast Distributed Optimization with Compression and Error Control Yuan Gao Rustem Islamov Sebastian U. Stich 39 8 0 06 Nov 2023
Compression with Exact Error Distribution for Federated Learning Mahmoud Hegazy Rémi Leluc Cheuk Ting Li Aymeric Dieuleveut FedML 18 9 0 31 Oct 2023
Convergence of Sign-based Random Reshuffling Algorithms for Nonconvex Optimization Zhen Qin Zhishuai Liu Pan Xu 31 1 0 24 Oct 2023
LASER: Linear Compression in Wireless Distributed Optimization Ashok Vardhan Makkuva Marco Bondaschi Thijs Vogels Martin Jaggi Hyeji Kim Michael C. Gastpar 87 3 0 19 Oct 2023