ATP: Adaptive Tensor Parallelism for Foundation Models

ATP: Adaptive Tensor Parallelism for Foundation Models

20 January 2023

Yang You

ArXiv (abs)PDF HTML

Papers citing "ATP: Adaptive Tensor Parallelism for Foundation Models"

5 / 5 papers shown

Title
HammingMesh: A Network Topology for Large-Scale Deep Learning Torsten Hoefler Tommaso Bonato Daniele De Sensi Salvatore Di Girolamo Shigang Li Marco Heddes Jon Belk Deepak Goel Miguel Castro Steve Scott 3DH GNN AI4CE 56 23 0 03 Sep 2022
Florence: A New Foundation Model for Computer Vision Lu Yuan Dongdong Chen Yi-Ling Chen Noel Codella Xiyang Dai ... Zhen Xiao Jianwei Yang Michael Zeng Luowei Zhou Pengchuan Zhang VLM 141 908 0 22 Nov 2021
Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training Yongbin Li Hongxin Liu Zhengda Bian Boxiang Wang Haichen Huang Fan Cui Chuan-Qing Wang Yang You GNN 82 147 0 28 Oct 2021
Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning Shaohua Wu Xudong Zhao Tong Yu Rongguo Zhang C. Shen ... Feng Li Hong Zhu Jiangang Luo Liang Xu Xuanwei Zhang ALM 51 60 0 10 Oct 2021
Horovod: fast and easy distributed deep learning in TensorFlow Alexander Sergeev Mike Del Balso 100 1,221 0 15 Feb 2018