AMSP: Reducing Communication Overhead of ZeRO for Efficient LLM Training

1 November 2023

Papers citing "AMSP: Reducing Communication Overhead of ZeRO for Efficient LLM Training"

3 / 3 papers shown

Title
Scaling Large Language Model Training on Frontier with Low-Bandwidth Partitioning Lang Xu Quentin G. Anthony Jacob Hatef Hari Subramoni Hari Subramoni Dhabaleswar K. Panda 44 0 0 08 Jan 2025
FP8-LM: Training FP8 Large Language Models Houwen Peng Kan Wu Yixuan Wei Guoshuai Zhao Yuxiang Yang ... Zheng-Wei Zhang Shuguang Liu Joe Chau Han Hu Peng Cheng MQ 59 40 0 27 Oct 2023
ZeRO++: Extremely Efficient Collective Communication for Giant Model Training Guanhua Wang Heyang Qin S. A. Jacobs Connor Holmes Samyam Rajbhandari Olatunji Ruwase Feng Yan Lei Yang Yuxiong He VLM 65 58 0 16 Jun 2023