Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training

28 October 2021

Yang You

Papers citing "Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training"

31 / 81 papers shown

Title
ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models Ziniu Li Tian Xu Yushun Zhang Zhihang Lin Yang Yu Ruoyu Sun Zhimin Luo 21 47 0 16 Oct 2023
TRANSOM: An Efficient Fault-Tolerant System for Training LLMs Baodong Wu Lei Xia Qingping Li Kangyu Li Xu Chen Yongqiang Guo Tieyao Xiang Yuheng Chen Shigang Li 32 11 0 16 Oct 2023
Rethinking Memory and Communication Cost for Efficient Large Language Model Training Chan Wu Hanxiao Zhang Lin Ju Jinjing Huang Youshao Xiao ... Siyuan Li Fanzhuang Meng Lei Liang Xiaolu Zhang Jun Zhou 20 4 0 09 Oct 2023
GrowLength: Accelerating LLMs Pretraining by Progressively Growing Training Length Hongye Jin Xiaotian Han Jingfeng Yang Zhimeng Jiang Chia-Yuan Chang Xia Hu 33 11 0 01 Oct 2023
RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Model Fengxiang Bie Yibo Yang Zhongzhu Zhou Adam Ghanem Minjia Zhang ... Pareesa Ameneh Golnari David A. Clifton Yuxiong He Dacheng Tao Shuaiwen Leon Song EGVM 30 18 0 02 Sep 2023
Reinforcement Learning with Human Feedback for Realistic Traffic Simulation Yulong Cao Boris Ivanovic Chaowei Xiao Marco Pavone 19 14 0 01 Sep 2023
Hanayo: Harnessing Wave-like Pipeline Parallelism for Enhanced Large Model Training Efficiency Ziming Liu Shenggan Cheng Hao Zhou Yang You 20 34 0 30 Aug 2023
Examining User-Friendly and Open-Sourced Large GPT Models: A Survey on Language, Multimodal, and Scientific GPT Models Kaiyuan Gao Su He Zhenyu He Jiacheng Lin Qizhi Pei Jie Shao Wei Zhang LM&MA SyDa 30 4 0 27 Aug 2023
A Comprehensive Overview of Large Language Models Humza Naveed Asad Ullah Khan Shi Qiu Muhammad Saqib Saeed Anwar Muhammad Usman Naveed Akhtar Nick Barnes Ajmal Saeed Mian OffRL 70 525 0 12 Jul 2023
Large Sequence Models for Sequential Decision-Making: A Survey Muning Wen Runji Lin Hanjing Wang Yaodong Yang Ying Wen Luo Mai Jun Wang Haifeng Zhang Weinan Zhang LM&Ro LRM 37 35 0 24 Jun 2023
Computron: Serving Distributed Deep Learning Models with Model Parallel Swapping Daniel Zou X. Jin Xueyang Yu Haotian Zhang J. Demmel MoE 26 0 0 24 Jun 2023
Blockwise Parallel Transformer for Large Context Models Hao Liu Pieter Abbeel 41 11 0 30 May 2023
Scaling Data-Constrained Language Models Niklas Muennighoff Alexander M. Rush Boaz Barak Teven Le Scao Aleksandra Piktus Nouamane Tazi S. Pyysalo Thomas Wolf Colin Raffel ALM 29 197 0 25 May 2023
Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline Zangwei Zheng Xiaozhe Ren Fuzhao Xue Yang Luo Xin Jiang Yang You 39 54 0 22 May 2023
DisCo-CLIP: A Distributed Contrastive Loss for Memory Efficient CLIP Training Yihao Chen Xianbiao Qi Jianan Wang Lei Zhang 18 16 0 17 Apr 2023
On Efficient Training of Large-Scale Deep Learning Models: A Literature Review Li Shen Yan Sun Zhiyuan Yu Liang Ding Xinmei Tian Dacheng Tao VLM 30 41 0 07 Apr 2023
Environment Transformer and Policy Optimization for Model-Based Offline Reinforcement Learning Pengqin Wang Meixin Zhu Shaojie Shen OffRL 25 1 0 07 Mar 2023
A Survey on Efficient Training of Transformers Bohan Zhuang Jing Liu Zizheng Pan Haoyu He Yuetian Weng Chunhua Shen 31 47 0 02 Feb 2023
ATP: Adaptive Tensor Parallelism for Foundation Models Shenggan Cheng Ziming Liu Jiangsu Du Yang You 21 6 0 20 Jan 2023
AutoDDL: Automatic Distributed Deep Learning with Near-Optimal Bandwidth Cost Jinfan Chen Shigang Li Ran Guo Jinhui Yuan Torsten Hoefler 23 2 0 17 Jan 2023
TAPS: Topology-Aware Intra-Operator Parallelism Strategy Searching Algorithm for Deep Neural Networks Peng Liang Hao Zheng Teng Su Linbo Qiao Dongsheng Li 14 0 0 11 Jan 2023
Protein Language Models and Structure Prediction: Connection and Progression Bozhen Hu Jun-Xiong Xia Jiangbin Zheng Cheng Tan Yufei Huang Yongjie Xu Stan Z. Li 21 40 0 30 Nov 2022
EnergonAI: An Inference System for 10-100 Billion Parameter Transformer Models Jiangsu Du Ziming Liu Jiarui Fang Shenggui Li Yongbin Li Yutong Lu Yang You MoE 27 4 0 06 Sep 2022
A Frequency-aware Software Cache for Large Recommendation System Embeddings Jiarui Fang Geng Zhang Jiatong Han Shenggui Li Zhengda Bian Yongbin Li Jin Liu Yang You 21 3 0 08 Aug 2022
Dive into Big Model Training Qinghua Liu Yuxiang Jiang MoMe AI4CE LRM 15 3 0 25 Jul 2022
Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Models Zhiquan Lai Shengwei Li Xudong Tang Ke-shi Ge Weijie Liu Yabo Duan Linbo Qiao Dongsheng Li 24 39 0 10 Jun 2022
PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Management Jiarui Fang Zilin Zhu Shenggui Li Hui Su Yang Yu Jie Zhou Yang You VLM 31 24 0 12 Aug 2021
Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines Shigang Li Torsten Hoefler GNN AI4CE LRM 77 131 0 14 Jul 2021
ZeRO-Offload: Democratizing Billion-Scale Model Training Jie Ren Samyam Rajbhandari Reza Yazdani Aminabadi Olatunji Ruwase Shuangyang Yang Minjia Zhang Dong Li Yuxiong He MoE 177 414 0 18 Jan 2021
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 280 2,015 0 28 Jul 2020
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 245 1,821 0 17 Sep 2019