v1v2 (latest)

DiLoCo: Distributed Low-Communication Training of Language Models

14 November 2023

Papers citing "DiLoCo: Distributed Low-Communication Training of Language Models"

26 / 26 papers shown

Title
NoLoCo: No-all-reduce Low Communication Training Method for Large Models Jari Kolehmainen Nikolay Blagoev John Donaghy Oğuzhan Ersoy Christopher Nies 110 0 0 12 Jun 2025
PC-MoE: Memory-Efficient and Privacy-Preserving Collaborative Training for Mixture-of-Experts LLMs Ze Yu Zhang Bolin Ding Bryan Kian Hsiang Low MoE 87 0 0 03 Jun 2025
Protocol Models: Scaling Decentralized Training with Communication-Efficient Model Parallelism Sameera Ramasinghe Thalaiyasingam Ajanthan Gil Avraham Yan Zuo Alexander Long GNN 95 0 0 02 Jun 2025
MuLoCo: Muon is a practical inner optimizer for DiLoCo Benjamin Thérien Xiaolong Huang Irina Rish Eugene Belilovsky MoE 51 0 0 29 May 2025
DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Models Alex Iacob Lorenzo Sani M. Safaryan Paris Giampouras Samuel Horváth ... Meghdad Kurmanji Preslav Aleksandrov William F. Shen Xinchi Qiu Nicholas D. Lane OffRL 104 0 0 28 May 2025
Pseudo-Asynchronous Local SGD: Robust and Efficient Data-Parallel Training Hiroki Naganuma Xinzhi Zhang Man-Chung Yue Ioannis Mitliagkas Philipp A. Witte Russell J. Hewett Yin Tat Lee 259 0 0 25 Apr 2025
Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo Zachary B. Charles Gabriel Teston Lucio Dery Keith Rush Nova Fallen Zachary Garrett Arthur Szlam Arthur Douillard 461 6 0 12 Mar 2025
SkipPipe: Partial and Reordered Pipelining Framework for Training LLMs in Heterogeneous Networks Nikolay Blagoev Lydia Yiyu Chen Oğuzhan Ersoy 99 2 0 27 Feb 2025
HDEE: Heterogeneous Domain Expert Ensemble Oğuzhan Ersoy Jari Kolehmainen Gabriel Passamani Andrade MoE 158 0 0 26 Feb 2025
Low-rank bias, weight decay, and model merging in neural networks Ilja Kuzborskij Yasin Abbasi-Yadkori 81 0 0 24 Feb 2025
EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Models Jialiang Cheng Ning Gao Yun Yue Zhiling Ye Jiadi Jiang Jian Sha OffRL 163 1 0 10 Dec 2024
Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Training Jared Fernandez Luca Wehrstedt Leonid Shamis Mostafa Elhoushi Kalyan Saladi Yonatan Bisk Emma Strubell Jacob Kahn 555 4 0 20 Nov 2024
Photon: Federated LLM Pre-Training Lorenzo Sani Alex Iacob Zeyu Cao Royson Lee Bill Marino ... Dongqi Cai Zexi Li Wanru Zhao Xinchi Qiu Nicholas D. Lane AI4CE 88 9 0 05 Nov 2024
Revisiting Reliability in Large-Scale Machine Learning Research Clusters Apostolos Kokolis Michael Kuchnik John Hoffman Adithya Kumar Parth Malani Faye Ma Zachary DeVito Siyang Song Kalyan Saladi Carole-Jean Wu 330 9 0 29 Oct 2024
DEPT: Decoupled Embeddings for Pre-training Language Models Alex Iacob Lorenzo Sani Meghdad Kurmanji William F. Shen Xinchi Qiu Dongqi Cai Yan Gao Nicholas D. Lane VLM 628 1 0 07 Oct 2024
No Need to Talk: Asynchronous Mixture of Language Models Anastasiia Filippova Angelos Katharopoulos David Grangier Ronan Collobert MoE 103 0 0 04 Oct 2024
Exploring Scaling Laws for Local SGD in Large Language Model Training Qiaozhi He Xiaomin Zhuang Zhihua Wu 92 4 0 20 Sep 2024
The AdEMAMix Optimizer: Better, Faster, Older Matteo Pagliardini Pierre Ablin David Grangier ODL 96 13 0 05 Sep 2024
WARP: On the Benefits of Weight Averaged Rewarded Policies Alexandre Ramé Johan Ferret Nino Vieillard Robert Dadashi Léonard Hussenot Pierre-Louis Cedoz Pier Giuseppe Sessa Sertan Girgin Arthur Douillard Olivier Bachem 136 23 0 24 Jun 2024
Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methods Tim Tsz-Kit Lau Weijian Li Chenwei Xu Han Liu Mladen Kolar 92 1 0 20 Jun 2024
Worldwide Federated Training of Language Models Alexandru Iacob Lorenzo Sani Bill Marino Preslav Aleksandrov William F. Shen Nicholas D. Lane FedML 86 2 0 23 May 2024
The Future of Large Language Model Pre-training is Federated Lorenzo Sani Alexandru Iacob Zeyu Cao Bill Marino Yan Gao ... Wanru Zhao William F. Shen Preslav Aleksandrov Xinchi Qiu Nicholas D. Lane AI4CE 161 21 0 17 May 2024
Simple and Scalable Strategies to Continually Pre-train Large Language Models Adam Ibrahim Benjamin Thérien Kshitij Gupta Mats L. Richter Quentin Anthony Timothée Lesort Eugene Belilovsky Irina Rish KELM CLL 109 63 0 13 Mar 2024
Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM Sainbayar Sukhbaatar O. Yu. Golovneva Vasu Sharma Hu Xu Xi Lin ... Jacob Kahn Shang-Wen Li Wen-tau Yih Jason Weston Xian Li MoMe OffRL MoE 98 69 0 12 Mar 2024
Training Neural Networks from Scratch with Parallel Low-Rank Adapters Minyoung Huh Brian Cheung Jeremy Bernstein Phillip Isola Pulkit Agrawal 106 12 0 26 Feb 2024
Asynchronous Local-SGD Training for Language Modeling Bo Liu Rachita Chhaparia Arthur Douillard Satyen Kale Andrei A. Rusu Jiajun Shen Arthur Szlam MarcÁurelio Ranzato FedML 93 11 0 17 Jan 2024