CO2: Efficient Distributed Training with Full Communication-Computation
Overlap

CO2: Efficient Distributed Training with Full Communication-Computation Overlap

29 January 2024

Zhen Qin

Papers citing "CO2: Efficient Distributed Training with Full Communication-Computation Overlap"

9 / 9 papers shown

Title
A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyond Xiaoye Qu Yafu Li Zhaochen Su Weigao Sun Jianhao Yan ... Chaochao Lu Yue Zhang Xian-Sheng Hua Bowen Zhou Yu Cheng ReLM OffRL LRM 85 14 0 27 Mar 2025
MoM: Linear Sequence Modeling with Mixture-of-Memories Jusen Du Weigao Sun Disen Lan Jiaxi Hu Yu-Xi Cheng KELM 75 3 0 19 Feb 2025
EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Models Jialiang Cheng Ning Gao Yun Yue Zhiling Ye Jiadi Jiang Jian Sha OffRL 77 0 0 10 Dec 2024
Distributed Sign Momentum with Local Steps for Training Transformers Shuhua Yu Ding Zhou Cong Xie An Xu Zhi-Li Zhang Xin Liu S. Kar 69 0 0 26 Nov 2024
ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training Adel Nabli Louis Fournier Pierre Erbacher Louis Serrano Eugene Belilovsky Edouard Oyallon FedML 46 1 0 03 Jun 2024
Linear Attention Sequence Parallelism Weigao Sun Zhen Qin Dong Li Xuyang Shen Yu Qiao Yiran Zhong 70 2 0 03 Apr 2024
MS-Net: A Multi-Path Sparse Model for Motion Prediction in Multi-Scenes Xiaqiang Tang Weigao Sun Siyuan Hu Yiyang Sun Yafeng Guo 43 5 0 01 Mar 2024
Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models Zhen Qin Weigao Sun Dong Li Xuyang Shen Weixuan Sun Yiran Zhong 69 22 0 09 Jan 2024
Feature Pyramid Networks for Object Detection Nayeon Lee Piotr Dollár Ross B. Girshick Kaiming He Bharath Hariharan Serge J. Belongie ObjD 183 21,813 0 09 Dec 2016