Tesseract: Parallelize the Tensor Parallelism Efficiently

v1v2 (latest)

Tesseract: Parallelize the Tensor Parallelism Efficiently

30 May 2021

Yang You

ArXiv (abs)PDF HTML

Papers citing "Tesseract: Parallelize the Tensor Parallelism Efficiently"

18 / 18 papers shown

Title
Model-Distributed Inference for Large Language Models at the Edge Davide Macario H. Seferoglu Erdem Koyuncu 37 0 0 13 May 2025
iServe: An Intent-based Serving System for LLMs Dimitrios Liakopoulos Tianrui Hu Prasoon Sinha N. Yadwadkar VLM 492 0 0 08 Jan 2025
ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning Samyam Rajbhandari Olatunji Ruwase Jeff Rasley Shaden Smith Yuxiong He GNN 83 385 0 16 Apr 2021
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai ... Matthias Minderer G. Heigold Sylvain Gelly Jakob Uszkoreit N. Houlsby ViT 667 41,369 0 22 Oct 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 826 42,332 0 28 May 2020
Scale MLPerf-0.6 models on Google TPU-v3 Pods Sameer Kumar Victor Bitorff Dehao Chen Chi-Heng Chou Blake A. Hechtman ... Peter Mattson Shibo Wang Tao Wang Yuanzhong Xu Zongwei Zhou 37 39 0 21 Sep 2019
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism Mohammad Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 331 1,914 0 17 Sep 2019
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes Yang You Jing Li Sashank J. Reddi Jonathan Hseu Sanjiv Kumar Srinadh Bhojanapalli Xiaodan Song J. Demmel Kurt Keutzer Cho-Jui Hsieh ODL 240 999 0 01 Apr 2019
Yet Another Accelerated SGD: ResNet-50 Training on ImageNet in 74.7 seconds Masafumi Yamazaki Akihiko Kasagi Akihiro Tabuchi Takumi Honda Masahiro Miwa Naoto Fukumoto Tsuguchika Tabaru Atsushi Ike Kohta Nakashima 39 88 0 29 Mar 2019
Image Classification at Supercomputer Scale Chris Ying Sameer Kumar Dehao Chen Tao Wang Youlong Cheng VLM 51 122 0 16 Nov 2018
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 1.8K 95,114 0 11 Oct 2018
Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes Xianyan Jia Shutao Song W. He Yangzihao Wang Haidong Rong ... Li Yu Tiegang Chen Guangxiao Hu Shaoshuai Shi Xiaowen Chu 77 384 0 30 Jul 2018
Mixed Precision Training Paulius Micikevicius Sharan Narang Jonah Alben G. Diamos Erich Elsen ... Boris Ginsburg Michael Houston Oleksii Kuchaiev Ganesh Venkatesh Hao Wu 166 1,804 0 10 Oct 2017
Large Batch Training of Convolutional Networks Yang You Igor Gitman Boris Ginsburg ODL 137 852 0 13 Aug 2017
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 722 132,199 0 12 Jun 2017
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour Priya Goyal Piotr Dollár Ross B. Girshick P. Noordhuis Lukasz Wesolowski Aapo Kyrola Andrew Tulloch Yangqing Jia Kaiming He 3DH 128 3,685 0 08 Jun 2017
Training Deep Nets with Sublinear Memory Cost Tianqi Chen Bing Xu Chiyuan Zhang Carlos Guestrin 103 1,171 0 21 Apr 2016
ImageNet Large Scale Visual Recognition Challenge Olga Russakovsky Jia Deng Hao Su J. Krause S. Satheesh ... A. Karpathy A. Khosla Michael S. Bernstein Alexander C. Berg Li Fei-Fei VLM ObjD 1.7K 39,590 0 01 Sep 2014