Automated Tensor Model Parallelism with Overlapped Communication for
Efficient Foundation Model Training

Automated Tensor Model Parallelism with Overlapped Communication for Efficient Foundation Model Training

25 May 2023

Xiaoge Deng

Dongsheng Li

Papers citing "Automated Tensor Model Parallelism with Overlapped Communication for Efficient Foundation Model Training"

10 / 10 papers shown

Title
Plexus: Taming Billion-edge Graphs with 3D Parallel GNN Training Aditya K. Ranjan Siddharth Singh Cunyang Wei A. Bhatele GNN 53 0 0 07 May 2025
Hiding Communication Cost in Distributed LLM Training via Micro-batch Co-execution Haiquan Wang Chaoyi Ruan Jia He Jiaqi Ruan Chengjie Tang Xiaosong Ma Cheng-rong Li 73 1 0 24 Nov 2024
Efficient Training of Large Language Models on Distributed Infrastructures: A Survey Jiangfei Duan Shuo Zhang Zerui Wang Lijuan Jiang Wenwen Qu ... Dahua Lin Yonggang Wen Xin Jin Tianwei Zhang Peng Sun 73 8 0 29 Jul 2024
Optimizing Large Model Training through Overlapped Activation Recomputation Ping Chen Wenjie Zhang Shuibing He Yingjie Gu Zhuwei Peng ... Yi Zheng Zhefeng Wang Yanlong Yin Gang Chen Gang Chen 35 5 0 13 Jun 2024
Lancet: Accelerating Mixture-of-Experts Training via Whole Graph Computation-Communication Overlapping Chenyu Jiang Ye Tian Zhen Jia Shuai Zheng Chuan Wu Yida Wang MoMe 29 7 0 30 Apr 2024
A 4D Hybrid Algorithm to Scale Parallel Training to Thousands of GPUs Siddharth Singh Prajwal Singhania Aditya K. Ranjan Zack Sating A. Bhatele 33 3 0 22 May 2023
PanGu-Σ: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing Xiaozhe Ren Pingyi Zhou Xinfan Meng Xinjing Huang Yadao Wang ... Jiansheng Wei Xin Jiang Teng Su Qun Liu Jun Yao ALM MoE 75 60 0 20 Mar 2023
ZeRO-Offload: Democratizing Billion-Scale Model Training Jie Ren Samyam Rajbhandari Reza Yazdani Aminabadi Olatunji Ruwase Shuangyang Yang Minjia Zhang Dong Li Yuxiong He MoE 177 414 0 18 Jan 2021
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 264 4,489 0 23 Jan 2020
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 245 1,821 0 17 Sep 2019