v1v2 (latest)

FreeRide: Harvesting Bubbles in Pipeline Parallelism

11 September 2024

Papers citing "FreeRide: Harvesting Bubbles in Pipeline Parallelism"

21 / 21 papers shown

Title
Towards Sustainable Large Language Model Serving Sophia Nguyen Beihao Zhou Yi Ding Sihang Liu 197 8 0 31 Dec 2024
DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency Jovan Stojkovic Chaojie Zhang Íñigo Goiri Josep Torrellas Esha Choukse 78 38 0 01 Aug 2024
Beyond the Bridge: Contention-Based Covert and Side Channel Attacks on Multi-GPU Interconnect Yicheng Zhang Ravan Nazaraliyev S. B. Dutta Nael B. Abu-Ghazaleh Andres Marquez Kevin Barker GNN 58 5 0 05 Apr 2024
ZeroPP: Unleashing Exceptional Parallelism Efficiency through Tensor-Parallelism-Free Methodology Ding Tang Lijuan Jiang Jiecheng Zhou Minxi Jin Hengjie Li Xingcheng Zhang Zhiling Pei Jidong Zhai 84 3 0 06 Feb 2024
G-Safe: Safe GPU Sharing in Multi-Tenant Environments Manos Pavlidakis Giorgos Vasiliadis Stelios Mavridis Anargyros Argyros A. Chazapis A. Bilas 25 3 0 17 Jan 2024
Zero Bubble Pipeline Parallelism Penghui Qi Xinyi Wan Guangxing Huang Min Lin 50 24 0 30 Nov 2023
Hanayo: Harnessing Wave-like Pipeline Parallelism for Enhanced Large Model Training Efficiency Ziming Liu Shenggan Cheng Hao Zhou Yang You 42 35 0 30 Aug 2023
PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matrices Kazuki Osawa Shigang Li Torsten Hoefler AI4CE 76 25 0 25 Nov 2022
Machine Learning Model Sizes and the Parameter Gap Pablo Villalobos J. Sevilla T. Besiroglu Lennart Heim A. Ho Marius Hobbhahn ALM ELM AI4CE 72 60 0 05 Jul 2022
OPT: Open Pre-trained Transformer Language Models Susan Zhang Stephen Roller Naman Goyal Mikel Artetxe Moya Chen ... Daniel Simig Punit Singh Koura Anjali Sridhar Tianlu Wang Luke Zettlemoyer VLM OSLM AI4CE 362 3,695 0 02 May 2022
Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs John Thorpe Pengzhan Zhao Jon Eyolfson Yifan Qiao Zhihao Jia Minjia Zhang Ravi Netravali Guoqing Harry Xu 61 58 0 26 Apr 2022
PaLM: Scaling Language Modeling with Pathways Aakanksha Chowdhery Sharan Narang Jacob Devlin Maarten Bosma Gaurav Mishra ... Kathy Meier-Hellstern Douglas Eck J. Dean Slav Petrov Noah Fiedel PILM LRM 529 6,293 0 05 Apr 2022
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model Shaden Smith M. Patwary Brandon Norick P. LeGresley Samyam Rajbhandari ... Mohammad Shoeybi Yuxiong He Michael Houston Saurabh Tiwary Bryan Catanzaro MoE 155 742 0 28 Jan 2022
VELTAIR: Towards High-Performance Multi-tenant Deep Learning Services via Adaptive Compilation and Scheduling Zihan Liu Jingwen Leng Zhihui Zhang Quan Chen Chao Li Minyi Guo 61 46 0 17 Jan 2022
Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines Shigang Li Torsten Hoefler GNN AI4CE LRM 118 136 0 14 Jul 2021
DAPPLE: A Pipelined Data Parallel Approach for Training Large Models Shiqing Fan Yi Rong Chen Meng Zongyan Cao Siyu Wang ... Jun Yang Lixue Xia Lansong Diao Xiaoyong Liu Wei Lin 94 239 0 02 Jul 2020
Memory-Efficient Pipeline-Parallel DNN Training Deepak Narayanan Amar Phanishayee Kaiyu Shi Xie Chen Matei A. Zaharia MoE 83 216 0 16 Jun 2020
torchgpipe: On-the-fly Pipeline Parallelism for Training Giant Models Chiheon Kim Heungsub Lee Myungryong Jeong Woonhyuk Baek Boogeon Yoon Ildoo Kim Sungbin Lim Sungwoong Kim MoE AI4CE 46 54 0 21 Apr 2020
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism Mohammad Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 336 1,917 0 17 Sep 2019
Mitigating Docker Security Issues R. Yasrab 13 26 0 13 Apr 2018
In-Datacenter Performance Analysis of a Tensor Processing Unit N. Jouppi C. Young Nishant Patil David Patterson Gaurav Agrawal ... Vijay Vasudevan Richard Walter Walter Wang Eric Wilcox Doe Hyun Yoon 237 4,644 0 16 Apr 2017