T3: Transparent Tracking & Triggering for Fine-grained Overlap of
Compute & Collectives

T3: Transparent Tracking & Triggering for Fine-grained Overlap of Compute & Collectives

30 January 2024

Suchita Pati

Shaizeen Aga

Mahzabeen Islam

Nuwan Jayasena

Matthew D. Sinclair

Papers citing "T3: Transparent Tracking & Triggering for Fine-grained Overlap of Compute & Collectives"

9 / 9 papers shown

Title
MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production Cheng Jin Ziheng Jiang Zhihao Bai Zheng Zhong Jing Liu ... Yanghua Peng Xuanzhe Liu Xuanzhe Liu Xin Jin Xin Liu MoE 14 0 0 16 May 2025
MSCCL++: Rethinking GPU Communication Abstractions for Cutting-edge AI Applications Aashaka Shah Abhinav Jangda Yangqiu Song Caio Rocha Changho Hwang ... Peng Cheng Qinghua Zhou Roshan Dathathri Saeed Maleki Ziyue Yang GNN 54 0 0 11 Apr 2025
Importance Sampling via Score-based Generative Models Heasung Kim Taekyun Lee Hyeji Kim Gustavo de Veciana MedIm DiffM 146 0 0 07 Feb 2025
Reducing the Cost of Dropout in Flash-Attention by Hiding RNG with GEMM Haiyue Ma Jian Liu Ronny Krashinsky 28 0 0 10 Oct 2024
ISO: Overlap of Computation and Communication within Seqenence For LLM Inference Bin Xiao Lei Su 28 0 0 04 Sep 2024
Efficient Training of Large Language Models on Distributed Infrastructures: A Survey Jiangfei Duan Shuo Zhang Zerui Wang Lijuan Jiang Wenwen Qu ... Dahua Lin Yonggang Wen Xin Jin Tianwei Zhang Peng Sun 73 8 0 29 Jul 2024
Optimizing Distributed ML Communication with Fused Computation-Collective Operations Kishore Punniyamurthy Khaled Hamidouche Bradford M. Beckmann FedML 34 8 0 11 May 2023
Scalable and Efficient MoE Training for Multitask Multilingual Models Young Jin Kim A. A. Awan Alexandre Muzio Andres Felipe Cruz Salinas Liyang Lu Amr Hendy Samyam Rajbhandari Yuxiong He Hany Awadalla MoE 104 84 0 22 Sep 2021
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 245 1,833 0 17 Sep 2019