v1v2v3 (latest)

Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training

5 December 2017

Song Han

ArXiv (abs)PDF HTML Github (222★)

Papers citing "Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training"

50 / 625 papers shown

Title
Spark Transformer: Reactivating Sparsity in FFN and Attention Chong You Kan Wu Zhipeng Jia Lin Chen Srinadh Bhojanapalli ... Felix X. Yu Prateek Jain David Culler Henry M. Levy Sanjiv Kumar 14 0 0 07 Jun 2025
Gradient Inversion Attacks on Parameter-Efficient Fine-Tuning Hasin Us Sami Swapneel Sen Amit K. Roy-Chowdhury S. Krishnamurthy Başak Güler FedML SILM 58 0 0 04 Jun 2025
Protocol Models: Scaling Decentralized Training with Communication-Efficient Model Parallelism Sameera Ramasinghe Thalaiyasingam Ajanthan Gil Avraham Yan Zuo Alexander Long GNN 69 0 0 02 Jun 2025
Memory-Efficient FastText: A Comprehensive Approach Using Double-Array Trie Structures and Mark-Compact Memory Management Yimin Du VLM 61 0 0 02 Jun 2025
DRAUN: An Algorithm-Agnostic Data Reconstruction Attack on Federated Unlearning Systems Hithem Lamri Manaar Alam Haiyan Jiang Michail Maniatakos MU 43 0 0 02 Jun 2025
SEMFED: Semantic-Aware Resource-Efficient Federated Learning for Heterogeneous NLP Tasks Sajid Hussain Muhammad Sohail Nauman Ali Khan FedML 11 0 0 26 May 2025
GraSS: Scalable Influence Function with Sparse Gradient Compression Pingbang Hu Joseph Melkonian Weijing Tang Han Zhao Jiaqi W. Ma TDI 278 0 0 25 May 2025
PacTrain: Pruning and Adaptive Sparse Gradient Compression for Efficient Collective Communication in Distributed Deep Learning Yisu Wang Ruilong Wu Xinjiao Li Dirk Kutscher 235 0 0 24 May 2025
ZenFlow: Enabling Stall-Free Offloading Training via Asynchronous Updates Tingfeng Lan Yusen Wu Bin Ma Zhaoyuan Su Rui Yang Tekin Bicer Dong Li Yue Cheng 218 0 0 18 May 2025
Communication-Efficient Wireless Federated Fine-Tuning for Large-Scale AI Models Bumjun Kim Wan Choi 88 0 0 01 May 2025
BackSlash: Rate Constrained Optimized Training of Large Language Models Jun Wu Jiangtao Wen Yuxing Han 138 1 0 23 Apr 2025
TAGC: Optimizing Gradient Communication in Distributed Transformer Training Igor Polyakov Alexey Dukhanov Egor Spirin 81 0 0 08 Apr 2025
FedSAUC: A Similarity-Aware Update Control for Communication-Efficient Federated Learning in Edge Computing Ming-Lun Lee Han-Chang Chou Yan-AnnChen FedML 97 6 0 07 Apr 2025
Towards Optimal Heterogeneous Client Sampling in Multi-Model Federated Learning Haoran Zhang Zejun Gong Zekai Li Marie Siew Carlee Joe-Wong Rachid El-Azouzi 142 0 0 07 Apr 2025
UniCoRN: Latent Diffusion-based Unified Controllable Image Restoration Network across Multiple Degradations Debabrata Mandal Soumitri Chattopadhyay Guansen Tong Praneeth Chakravarthula DiffM 103 1 0 20 Mar 2025
Trading-off Accuracy and Communication Cost in Federated Learning Mattia Jacopo Villani Emanuele Natale Frederik Mallmann-Trenn FedML 117 0 0 18 Mar 2025
Accelerated Distributed Optimization with Compression and Error Feedback Yuan Gao Anton Rodomanov Jeremy Rack Sebastian U. Stich 89 0 0 11 Mar 2025
Heterogeneity Matters even More in Distributed Learning: Study from Generalization Perspective Masoud Kavian Romain Chor Milad Sefidgaran Abdellatif Zaidi FedML 114 1 0 03 Mar 2025
Secure Aggregation in Federated Learning using Multiparty Homomorphic Encryption Erfan Hosseini Shuangyi Chen Ashish Khisti FedML 62 0 0 01 Mar 2025
Delayed Random Partial Gradient Averaging for Federated Learning Xinyi Hu FedML 113 0 0 31 Dec 2024
MARINA-P: Superior Performance in Non-smooth Federated Optimization with Adaptive Stepsizes Igor Sokolov Peter Richtárik 149 1 0 22 Dec 2024
Fed-AugMix: Balancing Privacy and Utility via Data Augmentation HaoYang Li Wei Chen Xiaojin Zhang FedML 102 0 0 18 Dec 2024
Adaptive Client Selection with Personalization for Communication Efficient Federated Learning Allan M. de Souza Filipe Maciel Joahannes B. D. da Costa L. Bittencourt E. Cerqueira A. A. F. Loureiro Leandro A. Villas FedML 154 20 0 26 Nov 2024
Lion Cub: Minimizing Communication Overhead in Distributed Lion Satoki Ishikawa Tal Ben-Nun B. Van Essen Rio Yokota Nikoli Dryden 122 0 0 25 Nov 2024
Sketched Adaptive Federated Deep Learning: A Sharp Convergence Analysis Zhijie Chen Qiaobo Li A. Banerjee FedML 114 0 0 11 Nov 2024
Optimal Defenses Against Gradient Reconstruction Attacks Yuxiao Chen Gamze Gürsoy Qi Lei FedML AAML 88 0 0 06 Nov 2024
$r$ Age- $k$ : Communication-Efficient Federated Learning Using Age Factor Matin Mortaheb Priyanka Kaswan S. Ulukus FedML 89 1 0 29 Oct 2024
Trustworthiness of Stochastic Gradient Descent in Distributed Learning Hongyang Li Caesar Wu Mohammed Chadli Said Mammar Pascal Bouvry 112 1 0 28 Oct 2024
LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics Thomas Robert M. Safaryan Ionut-Vlad Modoranu Dan Alistarh ODL 70 7 0 21 Oct 2024
FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression Zhenheng Tang Xueze Kang Yiming Yin Xinglin Pan Yuxin Wang ... Shaohuai Shi Amelie Chi Zhou Bo Li Bingsheng He Xiaowen Chu AI4CE 109 8 0 16 Oct 2024
A Comprehensive Survey on Joint Resource Allocation Strategies in Federated Edge Learning Jingbo Zhang Qiong Wu Pingyi Fan Nan Cheng FedML 75 0 0 10 Oct 2024
Private and Communication-Efficient Federated Learning based on Differentially Private Sketches Meifan Zhang Zhanhong Xie Lihua Yin FedML 54 1 0 08 Oct 2024
No Need to Talk: Asynchronous Mixture of Language Models Anastasiia Filippova Angelos Katharopoulos David Grangier Ronan Collobert MoE 91 0 0 04 Oct 2024
Temporal Predictive Coding for Gradient Compression in Distributed Learning Adrian Edin Zheng Chen Michel Kieffer Mikael Johansson 51 1 0 03 Oct 2024
Novel Gradient Sparsification Algorithm via Bayesian Inference Ali Bereyhi B. Liang G. Boudreau Ali Afana 61 2 0 23 Sep 2024
Data Poisoning and Leakage Analysis in Federated Learning Wenqi Wei Tiansheng Huang Zachary Yahn Anoop Singhal Margaret Loper Ling Liu FedML SILM 40 0 0 19 Sep 2024
Advances in APPFL: A Comprehensive and Extensible Federated Learning Framework Zilinghan Li Shilan He Ze Yang Minseok Ryu Kibaek Kim Ravi K. Madduri FedML 150 14 0 17 Sep 2024
S-STE: Continuous Pruning Function for Efficient 2:4 Sparse Pre-training Yuezhou Hu Jun-Jie Zhu Jianfei Chen 126 0 0 13 Sep 2024
Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models Yao Shu Wenyang Hu Szu Hui Ng Bryan Kian Hsiang Low Fei Richard Yu FedML 103 2 0 10 Sep 2024
FedFQ: Federated Learning with Fine-Grained Quantization Haowei Li Weiying Xie Hangyu Ye Jitao Ma Shuran Ma Yunsong Li FedML MQ 41 1 0 16 Aug 2024
Rina: Enhancing Ring-AllReduce with In-network Aggregation in Distributed Model Training Zixuan Chen Xuandong Liu Minglin Li Yinfan Hu Hao Mei Huifeng Xing Hao Wang Wanxin Shi Sen Liu Yang Xu 72 1 0 29 Jul 2024
Ordered Momentum for Asynchronous SGD Chang-Wei Shi Yi-Rui Yang Wu-Jun Li ODL 169 0 0 27 Jul 2024
Toward INT4 Fixed-Point Training via Exploring Quantization Error for Gradients Dohyung Kim Junghyup Lee Jeimin Jeon Jaehyeon Moon Bumsub Ham MQ 88 0 0 17 Jul 2024
Scalable and Reliable Over-the-Air Federated Edge Learning Maximilian Egger Christoph Hofmeister Cem Kaya Rawad Bitar Antonia Wachter-Zeh 77 1 0 16 Jul 2024
CCVA-FL: Cross-Client Variations Adaptive Federated Learning for Medical Imaging Sunny Gupta Amit Sethi MedIm 56 0 0 16 Jul 2024
Foundation Model Engineering: Engineering Foundation Models Just as Engineering Software Dezhi Ran Mengzhou Wu Wei Yang Tao Xie AI4CE 75 2 0 11 Jul 2024
A Joint Approach to Local Updating and Gradient Compression for Efficient Asynchronous Federated Learning Jiajun Song Jiajun Luo Rongwei Lu Shuzhao Xie Bin Chen Zhi Wang FedML 49 0 0 06 Jul 2024
VcLLM: Video Codecs are Secretly Tensor Codecs Ceyu Xu Yongji Wu Xinyu Yang Beidi Chen Matthew Lentz Danyang Zhuo Lisa Wu Wills 102 0 0 29 Jun 2024
FedAQ: Communication-Efficient Federated Edge Learning via Joint Uplink and Downlink Adaptive Quantization Linping Qu Shenghui Song Chi-Ying Tsui MQ FedML 53 4 0 26 Jun 2024
Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradients Aashiq Muhamed Oscar Li David Woodruff Mona Diab Virginia Smith 94 13 0 25 Jun 2024