Achieving Dimension-Free Communication in Federated Learning via Zeroth-Order Optimization

24 May 2024

Papers citing "Achieving Dimension-Free Communication in Federated Learning via Zeroth-Order Optimization"

50 / 59 papers shown

Title
Exact and Linear Convergence for Federated Learning under Arbitrary Client Participation is Attainable Bicheng Ying Zhe Li Haibo Yang FedML 87 0 0 25 Mar 2025
Efficient Zero-Order Federated Finetuning of Language Models for Resource-Constrained Devices Mohamed Aboelenien Ahmed Kilian Pfeiffer R. Khalili Heba Khdr J. Henkel FedML 127 0 0 17 Feb 2025
FedComLoc: Communication-Efficient Distributed Training of Sparse and Quantized Models Kai Yi Georg Meinhardt Laurent Condat Peter Richtárik FedML 67 6 0 14 Mar 2024
Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark Yihua Zhang Pingzhi Li Junyuan Hong Jiaxiang Li Yimeng Zhang ... Wotao Yin Mingyi Hong Zhangyang Wang Sijia Liu Tianlong Chen 76 48 0 18 Feb 2024
On the Convergence of Zeroth-Order Federated Tuning for Large Language Models Zhenqing Ling Daoyuan Chen Liuyi Yao Yaliang Li Ying Shen FedML 76 13 0 08 Feb 2024
Private Fine-tuning of Large Language Models with Zeroth-order Optimization Xinyu Tang Ashwinee Panda Milad Nasr Saeed Mahloujifar Prateek Mittal 112 20 0 09 Jan 2024
Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning Bingchen Zhao Haoqin Tu Chen Wei Jieru Mei Cihang Xie 59 33 0 18 Dec 2023
DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Training Aochuan Chen Yimeng Zhang Jinghan Jia James Diffenderfer Jiancheng Liu Konstantinos Parasyris Yihua Zhang Zheng Zhang B. Kailkhura Sijia Liu 86 45 0 03 Oct 2023
CORE: Common Random Reconstruction for Distributed Optimization with Provable Low Communication Complexity Pengyun Yue Hanzheng Zhao Cong Fang Di He Liwei Wang Zhouchen Lin Song-Chun Zhu 52 1 0 23 Sep 2023
Stochastic Controlled Averaging for Federated Learning with Communication Compression Xinmeng Huang Ping Li Xiaoyun Li 53 201 0 16 Aug 2023
Fine-Tuning Language Models with Just Forward Passes Sadhika Malladi Tianyu Gao Eshaan Nichani Alexandru Damian Jason D. Lee Danqi Chen Sanjeev Arora 103 186 0 27 May 2023
Communication-Efficient Federated Learning for Heterogeneous Edge Devices Based on Adaptive Gradient Quantization Heting Liu Fang He Guohong Cao FedML MQ 63 24 0 16 Dec 2022
Communication-Efficient Federated Learning With Data and Client Heterogeneity Hossein Zakerinia Shayan Talaei Giorgi Nadiradze Dan Alistarh FedML 58 8 0 20 Jun 2022
Communication-Efficient Adaptive Federated Learning Yujia Wang Lu Lin Jinghui Chen FedML 47 71 0 05 May 2022
OPT: Open Pre-trained Transformer Language Models Susan Zhang Stephen Roller Naman Goyal Mikel Artetxe Moya Chen ... Daniel Simig Punit Singh Koura Anjali Sridhar Tianlu Wang Luke Zettlemoyer VLM OSLM AI4CE 286 3,583 0 02 May 2022
Wireless Quantized Federated Learning: A Joint Computation and Communication Design Pavlos S. Bouzinis P. Diamantoulakis G. Karagiannidis MQ FedML 51 31 0 11 Mar 2022
Communication-Efficient Stochastic Zeroth-Order Optimization for Federated Learning Wenzhi Fang Ziyi Yu Yuning Jiang Yuanming Shi Colin N. Jones Yong Zhou FedML 94 58 0 24 Jan 2022
DAdaQuant: Doubly-adaptive quantization for communication-efficient Federated Learning Robert Hönig Yiren Zhao Robert D. Mullins FedML 135 56 0 31 Oct 2021
Multitask Prompted Training Enables Zero-Shot Task Generalization Victor Sanh Albert Webson Colin Raffel Stephen H. Bach Lintang Sutawika ... T. Bers Stella Biderman Leo Gao Thomas Wolf Alexander M. Rush LRM 315 1,679 0 15 Oct 2021
Asynchronous Distributed Reinforcement Learning for LQR Control via Zeroth-Order Block Coordinate Descent Gangshan Jing H. Bai Jemin George A. Chakrabortty P. Sharma 43 8 0 26 Jul 2021
A Field Guide to Federated Optimization Jianyu Wang Zachary B. Charles Zheng Xu Gauri Joshi H. B. McMahan ... Mi Zhang Tong Zhang Chunxiang Zheng Chen Zhu Wennan Zhu FedML 257 415 0 14 Jul 2021
LoRA: Low-Rank Adaptation of Large Language Models J. E. Hu Yelong Shen Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang Weizhu Chen OffRL AI4TS AI4CE ALM AIMat 312 10,099 0 17 Jun 2021
CFedAvg: Achieving Efficient Communication and Fast Convergence in Non-IID Federated Learning Haibo Yang Jia Liu Elizabeth S. Bentley FedML 24 18 0 14 Jun 2021
Federated Learning Meets Blockchain in Edge Computing: Opportunities and Challenges Dinh C. Nguyen Ming Ding Quoc-Viet Pham P. Pathirana Long Bao Jun Seneviratne Jun Li Dusit Niyato Life Fellow Ieee Poor FedML 66 426 0 05 Apr 2021
A Zeroth-Order Block Coordinate Descent Algorithm for Huge-Scale Black-Box Optimization HanQin Cai Y. Lou Daniel McKenzie W. Yin 65 43 0 21 Feb 2021
Adaptive Quantization of Model Updates for Communication-Efficient Federated Learning Divyansh Jhunjhunwala Advait Gadhikar Gauri Joshi Yonina C. Eldar FedML MQ 41 108 0 08 Feb 2021
Time-Correlated Sparsification for Communication-Efficient Federated Learning Emre Ozfatura Kerem Ozfatura Deniz Gunduz FedML 52 48 0 21 Jan 2021
Prefix-Tuning: Optimizing Continuous Prompts for Generation Xiang Lisa Li Percy Liang 184 4,209 0 01 Jan 2021
Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks Yikai Wu Xingyu Zhu Chenwei Wu Annie Wang Rong Ge 58 44 0 08 Oct 2020
Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra Vardan Papyan 38 78 0 27 Aug 2020
Communication-Efficient Federated Learning via Optimal Client Sampling Mónica Ribero H. Vikalo FedML 50 94 0 30 Jul 2020
Federated Learning with Compression: Unified Analysis and Sharp Guarantees Farzin Haddadpour Mohammad Mahdi Kamani Aryan Mokhtari M. Mahdavi FedML 63 275 0 02 Jul 2020
A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learning Sijia Liu Pin-Yu Chen B. Kailkhura Gaoyuan Zhang A. Hero III P. Varshney 63 228 0 11 Jun 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 544 41,106 0 28 May 2020
A Review of Winograd Schema Challenge Datasets and Approaches Vid Kocijan Thomas Lukasiewicz E. Davis G. Marcus L. Morgenstern 58 44 0 23 Apr 2020
Adaptive Gradient Sparsification for Efficient Federated Learning: An Online Learning Approach Pengchao Han Shiqiang Wang K. Leung FedML 53 178 0 14 Jan 2020
Distributed Reinforcement Learning for Decentralized Linear Quadratic Control: A Derivative-Free Policy Optimization Approach Yingying Li Yujie Tang Runyu Zhang Na Li 50 101 0 19 Dec 2019
PyHessian: Neural Networks Through the Lens of the Hessian Z. Yao A. Gholami Kurt Keutzer Michael W. Mahoney ODL 45 294 0 16 Dec 2019
Advances and Open Problems in Federated Learning Peter Kairouz H. B. McMahan Brendan Avent A. Bellet M. Bennis ... Zheng Xu Qiang Yang Felix X. Yu Han Yu Sen Zhao FedML AI4CE 150 6,177 0 10 Dec 2019
Understanding Top-k Sparsification in Distributed Deep Learning Shaoshuai Shi Xiaowen Chu Ka Chun Cheung Simon See 175 96 0 20 Nov 2019
ZO-AdaMM: Zeroth-Order Adaptive Momentum Method for Black-Box Optimization Xiangyi Chen Sijia Liu Kaidi Xu Xingguo Li Xue Lin Mingyi Hong David Cox ODL 58 106 0 15 Oct 2019
FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization Amirhossein Reisizadeh Aryan Mokhtari Hamed Hassani Ali Jadbabaie Ramtin Pedarsani FedML 233 767 0 28 Sep 2019
PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization Thijs Vogels Sai Praneeth Karimireddy Martin Jaggi 64 320 0 31 May 2019
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions Christopher Clark Kenton Lee Ming-Wei Chang Tom Kwiatkowski Michael Collins Kristina Toutanova 191 1,475 0 24 May 2019
An Investigation into Neural Net Optimization via Hessian Eigenvalue Density Behrooz Ghorbani Shankar Krishnan Ying Xiao ODL 60 320 0 29 Jan 2019
Rethinking the Value of Network Pruning Zhuang Liu Mingjie Sun Tinghui Zhou Gao Huang Trevor Darrell 36 1,460 0 11 Oct 2018
Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning Charles H. Martin Michael W. Mahoney AI4CE 92 196 0 02 Oct 2018
Sparsified SGD with Memory Sebastian U. Stich Jean-Baptiste Cordonnier Martin Jaggi 66 743 0 20 Sep 2018
WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations Mohammad Taher Pilehvar Jose Camacho-Collados 142 478 0 28 Aug 2018
Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimization Sijia Liu B. Kailkhura Pin-Yu Chen Pai-Shun Ting Shiyu Chang Lisa Amini 87 178 0 25 May 2018