SkipPipe: Partial and Reordered Pipelining Framework for Training LLMs in Heterogeneous Networks

27 February 2025

Papers citing "SkipPipe: Partial and Reordered Pipelining Framework for Training LLMs in Heterogeneous Networks"

7 / 7 papers shown

Title
Understanding Robustness of Transformers for Image Classification Srinadh Bhojanapalli Ayan Chakrabarti Daniel Glasner Daliang Li Thomas Unterthiner Andreas Veit ViT 38 382 0 26 Mar 2021
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 432 41,106 0 28 May 2020
HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism Jay H. Park Gyeongchan Yun Chang Yi N. T. Nguyen Seungmin Lee Jaesik Choi S. Noh Young-ri Choi MoE 35 130 0 28 May 2020
Reducing Transformer Depth on Demand with Structured Dropout Angela Fan Edouard Grave Armand Joulin 86 586 0 25 Sep 2019
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism Mohammad Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 281 1,861 0 17 Sep 2019
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 338 129,831 0 12 Jun 2017
Deep Networks with Stochastic Depth Gao Huang Yu Sun Zhuang Liu Daniel Sedra Kilian Q. Weinberger 134 2,344 0 30 Mar 2016