Fine-tuning Language Models over Slow Networks using Activation
Compression with Guarantees

v1v2v3 (latest)

Fine-tuning Language Models over Slow Networks using Activation Compression with Guarantees

2 June 2022

Christopher Ré

ArXiv (abs)PDF HTML

Papers citing "Fine-tuning Language Models over Slow Networks using Activation Compression with Guarantees"

10 / 10 papers shown

Title
Accelerating AllReduce with a Persistent Straggler Arjun Devraj Eric Ding Abhishek Vijaya Kumar Robert Kleinberg Rachee Singh 56 0 0 29 May 2025
COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Training Haocheng Xi Han Cai Ligeng Zhu Yaojie Lu Kurt Keutzer Jianfei Chen Song Han MQ 173 11 0 25 Oct 2024
Hazards from Increasingly Accessible Fine-Tuning of Downloadable Foundation Models Alan Chan Ben Bucknall Herbie Bradley David M. Krueger 63 6 0 22 Dec 2023
Distributed Inference and Fine-tuning of Large Language Models Over The Internet Alexander Borzunov Max Ryabinin Artem Chumachenko Dmitry Baranchuk Tim Dettmers Younes Belkada Pavel Samygin Colin Raffel MoE ALM 65 42 0 13 Dec 2023
Fast Distributed Inference Serving for Large Language Models Bingyang Wu Yinmin Zhong Zili Zhang Gang Huang Xuanzhe Liu Xin Jin 88 102 0 10 May 2023
Scaling Expert Language Models with Unsupervised Domain Discovery Suchin Gururangan Margaret Li M. Lewis Weijia Shi Tim Althoff Noah A. Smith Luke Zettlemoyer MoE 102 48 0 24 Mar 2023
Quantized Distributed Training of Large Models with Convergence Guarantees I. Markov Adrian Vladu Qi Guo Dan Alistarh MQ 97 11 0 05 Feb 2023
Does compressing activations help model parallel training? S. Bian Dacheng Li Hongyi Wang Eric P. Xing Shivaram Venkataraman 74 9 0 06 Jan 2023
RCD-SGD: Resource-Constrained Distributed SGD in Heterogeneous Environment via Submodular Partitioning Haoze He Parijat Dube 58 1 0 02 Nov 2022
lo-fi: distributed fine-tuning without communication Mitchell Wortsman Suchin Gururangan Shen Li Ali Farhadi Ludwig Schmidt Michael G. Rabbat Ari S. Morcos 108 24 0 19 Oct 2022

We use cookies and other tracking technologies to improve your browsing experience on our website, to show you personalized content and targeted ads, to analyze our website traffic, and to understand where our visitors are coming from. See our policy.