Fine-tuning Language Models over Slow Networks using Activation
Compression with Guarantees

Fine-tuning Language Models over Slow Networks using Activation Compression with Guarantees

2 June 2022

Christopher Ré

Papers citing "Fine-tuning Language Models over Slow Networks using Activation Compression with Guarantees"

13 / 13 papers shown

Title
COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Training Haocheng Xi Han Cai Ligeng Zhu Yaojie Lu Kurt Keutzer Jianfei Chen Song Han MQ 75 9 0 25 Oct 2024
Hazards from Increasingly Accessible Fine-Tuning of Downloadable Foundation Models Alan Chan Ben Bucknall Herbie Bradley David M. Krueger 14 6 0 22 Dec 2023
Distributed Inference and Fine-tuning of Large Language Models Over The Internet Alexander Borzunov Max Ryabinin Artem Chumachenko Dmitry Baranchuk Tim Dettmers Younes Belkada Pavel Samygin Colin Raffel MoE ALM 15 39 0 13 Dec 2023
Fast Distributed Inference Serving for Large Language Models Bingyang Wu Yinmin Zhong Zili Zhang Gang Huang Xuanzhe Liu Xin Jin 30 92 0 10 May 2023
Scaling Expert Language Models with Unsupervised Domain Discovery Suchin Gururangan Margaret Li M. Lewis Weijia Shi Tim Althoff Noah A. Smith Luke Zettlemoyer MoE 25 46 0 24 Mar 2023
Quantized Distributed Training of Large Models with Convergence Guarantees I. Markov Adrian Vladu Qi Guo Dan Alistarh MQ 31 11 0 05 Feb 2023
Does compressing activations help model parallel training? S. Bian Dacheng Li Hongyi Wang Eric P. Xing Shivaram Venkataraman 19 5 0 06 Jan 2023
RCD-SGD: Resource-Constrained Distributed SGD in Heterogeneous Environment via Submodular Partitioning Haoze He Parijat Dube 15 1 0 02 Nov 2022
lo-fi: distributed fine-tuning without communication Mitchell Wortsman Suchin Gururangan Shen Li Ali Farhadi Ludwig Schmidt Michael G. Rabbat Ari S. Morcos 32 24 0 19 Oct 2022
ScaleCom: Scalable Sparsified Gradient Compression for Communication-Efficient Distributed Training Chia-Yu Chen Jiamin Ni Songtao Lu Xiaodong Cui Pin-Yu Chen ... Naigang Wang Swagath Venkataramani Vijayalakshmi Srinivasan Wei Zhang K. Gopalakrishnan 27 18 0 21 Apr 2021
Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks Torsten Hoefler Dan Alistarh Tal Ben-Nun Nikoli Dryden Alexandra Peste MQ 141 684 0 31 Jan 2021
An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systems A. Abdelmoniem Ahmed Elzanaty Mohamed-Slim Alouini Marco Canini 51 74 0 26 Jan 2021
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 245 1,821 0 17 Sep 2019