v1v2 (latest)

DELTA: Dynamically Optimizing GPU Memory beyond Tensor Recomputation

30 March 2022

Yuliang Liu

Dongsheng Li

Papers citing "DELTA: Dynamically Optimizing GPU Memory beyond Tensor Recomputation"

37 / 37 papers shown

Title
AI and Memory Wall A. Gholami Z. Yao Sehoon Kim Coleman Hooper Michael W. Mahoney Kurt Keutzer 72 158 0 21 Mar 2024
ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning Samyam Rajbhandari Olatunji Ruwase Jeff Rasley Shaden Smith Yuxiong He GNN 83 387 0 16 Apr 2021
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows Ze Liu Yutong Lin Yue Cao Han Hu Yixuan Wei Zheng Zhang Stephen Lin B. Guo ViT 463 21,564 0 25 Mar 2021
ZeRO-Offload: Democratizing Billion-Scale Model Training Jie Ren Samyam Rajbhandari Reza Yazdani Aminabadi Olatunji Ruwase Shuangyang Yang Minjia Zhang Dong Li Yuxiong He MoE 262 429 0 18 Jan 2021
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity W. Fedus Barret Zoph Noam M. Shazeer MoE 88 2,220 0 11 Jan 2021
Dynamic Tensor Rematerialization Marisa Kirisame Steven Lyubomirsky Altan Haan Jennifer Brennan Mike He Jared Roesch Tianqi Chen Zachary Tatlock 80 93 0 17 Jun 2020
Training Large Neural Networks with Constant Memory using a New Execution Algorithm B. Pudipeddi Maral Mesmakhosroshahi Jinwen Xi S. Bharadwaj 46 57 0 13 Feb 2020
PyTorch: An Imperative Style, High-Performance Deep Learning Library Adam Paszke Sam Gross Francisco Massa Adam Lerer James Bradbury ... Sasank Chilamkurthy Benoit Steiner Lu Fang Junjie Bai Soumith Chintala ODL 529 42,559 0 03 Dec 2019
Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerialization Paras Jain Ajay Jain Aniruddha Nrusimha A. Gholami Pieter Abbeel Kurt Keutzer Ion Stoica Joseph E. Gonzalez 64 195 0 07 Oct 2019
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 1.8K 95,175 0 11 Oct 2018
PipeDream: Fast and Efficient Pipeline Parallel DNN Training A. Harlap Deepak Narayanan Amar Phanishayee Vivek Seshadri Nikhil R. Devanur G. Ganger Phillip B. Gibbons AI4CE 61 254 0 08 Jun 2018
i-RevNet: Deep Invertible Networks J. Jacobsen A. Smeulders Edouard Oyallon 83 333 0 20 Feb 2018
SuperNeurons: Dynamic GPU Memory Management for Training Deep Neural Networks Linnan Wang Jinmian Ye Yiyang Zhao Wei Wu Ang Li Shuaiwen Leon Song Zenglin Xu Tim Kraska 3DH 101 267 0 13 Jan 2018
In-Place Activated BatchNorm for Memory-Optimized Training of DNNs Samuel Rota Buló Lorenzo Porzi Peter Kontschieder 65 357 0 07 Dec 2017
Nonparametric regression using deep neural networks with ReLU activation function Johannes Schmidt-Hieber 230 815 0 22 Aug 2017
The Reversible Residual Network: Backpropagation Without Storing Activations Aidan Gomez Mengye Ren R. Urtasun Roger C. Grosse 74 551 0 14 Jul 2017
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices Xiangyu Zhang Xinyu Zhou Mengxiao Lin Jian Sun AI4TS 144 6,884 0 04 Jul 2017
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 730 132,363 0 12 Jun 2017
Compressing DMA Engine: Leveraging Activation Sparsity for Training Deep Neural Networks Minsoo Rhu Mike O'Connor Niladrish Chatterjee Jeff Pool S. Keckler 65 177 0 03 May 2017
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications Andrew G. Howard Menglong Zhu Bo Chen Dmitry Kalenichenko Weijun Wang Tobias Weyand M. Andreetto Hartwig Adam 3DH 1.2K 20,880 0 17 Apr 2017
Aggregated Residual Transformations for Deep Neural Networks Saining Xie Ross B. Girshick Piotr Dollár Zhuowen Tu Kaiming He 522 10,345 0 16 Nov 2016
SQuAD: 100,000+ Questions for Machine Comprehension of Text Pranav Rajpurkar Jian Zhang Konstantin Lopyrev Percy Liang RALM 312 8,169 0 16 Jun 2016
TensorFlow: A system for large-scale machine learning Martín Abadi P. Barham Jianmin Chen Zhiwen Chen Andy Davis ... Vijay Vasudevan Pete Warden Martin Wicke Yuan Yu Xiaoqiang Zhang GNN AI4CE 433 18,361 0 27 May 2016
Training Deep Nets with Sublinear Memory Cost Tianqi Chen Bing Xu Chiyuan Zhang Carlos Guestrin 106 1,171 0 21 Apr 2016
Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning Christian Szegedy Sergey Ioffe Vincent Vanhoucke Alexander A. Alemi 381 14,260 0 23 Feb 2016
EIE: Efficient Inference Engine on Compressed Deep Neural Network Song Han Xingyu Liu Huizi Mao Jing Pu A. Pedram M. Horowitz W. Dally 127 2,459 0 04 Feb 2016
Deep Residual Learning for Image Recognition Kaiming He Xinming Zhang Shaoqing Ren Jian Sun MedIm 2.2K 194,322 0 10 Dec 2015
MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems Tianqi Chen Mu Li Yutian Li Min Lin Naiyan Wang Minjie Wang Tianjun Xiao Bing Xu Chiyuan Zhang Zheng Zhang 196 2,248 0 03 Dec 2015
BinaryConnect: Training Deep Neural Networks with binary weights during propagations Matthieu Courbariaux Yoshua Bengio J. David MQ 212 2,992 0 02 Nov 2015
Learning both Weights and Connections for Efficient Neural Networks Song Han Jeff Pool J. Tran W. Dally CVBM 313 6,694 0 08 Jun 2015
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks Shaoqing Ren Kaiming He Ross B. Girshick Jian Sun AIMat ObjD 525 62,360 0 04 Jun 2015
U-Net: Convolutional Networks for Biomedical Image Segmentation Olaf Ronneberger Philipp Fischer Thomas Brox SSeg 3DV 1.9K 77,341 0 18 May 2015
Fast R-CNN Ross B. Girshick ObjD 309 25,081 0 30 Apr 2015
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift Sergey Ioffe Christian Szegedy OOD 465 43,341 0 11 Feb 2015
Deep Learning with Limited Numerical Precision Suyog Gupta A. Agrawal K. Gopalakrishnan P. Narayanan HAI 207 2,049 0 09 Feb 2015
Adam: A Method for Stochastic Optimization Diederik P. Kingma Jimmy Ba ODL 2.0K 150,312 0 22 Dec 2014
Compressing Deep Convolutional Networks using Vector Quantization Yunchao Gong Liu Liu Ming Yang Lubomir D. Bourdev MQ 168 1,171 0 18 Dec 2014