Dynamic Tensor Rematerialization

Dynamic Tensor Rematerialization

17 June 2020

Marisa Kirisame

Steven Lyubomirsky

Jennifer Brennan

Zachary Tatlock

Papers citing "Dynamic Tensor Rematerialization"

14 / 14 papers shown

Title
GPU Memory Usage Optimization for Backward Propagation in Deep Network Training Ding-Yong Hong Tzu-Hsien Tsai Ning Wang Pangfeng Liu Jan-Jan Wu 44 0 0 18 Feb 2025
QFT: Quantized Full-parameter Tuning of LLMs with Affordable Resources Zhikai Li Xiaoxuan Liu Banghua Zhu Zhen Dong Qingyi Gu Kurt Keutzer MQ 32 7 0 11 Oct 2023
Aggregating Capacity in FL through Successive Layer Training for Computationally-Constrained Devices Kilian Pfeiffer R. Khalili J. Henkel FedML 44 5 0 26 May 2023
Automated Tensor Model Parallelism with Overlapped Communication for Efficient Foundation Model Training Shengwei Li Zhiquan Lai Yanqi Hao Weijie Liu Ke-shi Ge Xiaoge Deng Dongsheng Li KaiCheng Lu 16 10 0 25 May 2023
An Evaluation of Memory Optimization Methods for Training Neural Networks Xiaoxuan Liu Siddharth Jha Alvin Cheung 29 0 0 26 Mar 2023
Colossal-Auto: Unified Automation of Parallelization and Activation Checkpoint for Large-scale Models Yuliang Liu Shenggui Li Jiarui Fang Yan Shao Boyuan Yao Yang You OffRL 27 7 0 06 Feb 2023
On-device Training: A First Overview on Existing Systems Shuai Zhu Thiemo Voigt Jeonggil Ko Fatemeh Rahimian 34 14 0 01 Dec 2022
Mimose: An Input-Aware Checkpointing Planner for Efficient Training on GPU Jian-He Liao Mingzhen Li Qingxiao Sun Jiwei Hao F. Yu ... Ye Tao Zicheng Zhang Hailong Yang Zhongzhi Luan D. Qian 23 4 0 06 Sep 2022
POET: Training Neural Networks on Tiny Devices with Integrated Rematerialization and Paging Shishir G. Patil Paras Jain P. Dutta Ion Stoica Joseph E. Gonzalez 12 35 0 15 Jul 2022
DELTA: Dynamically Optimizing GPU Memory beyond Tensor Recomputation Yu Tang Chenyu Wang Yufan Zhang Yuliang Liu Xingcheng Zhang Linbo Qiao Zhiquan Lai Dongsheng Li 21 4 0 30 Mar 2022
Survey on Large Scale Neural Network Training Julia Gusak Daria Cherniuk Alena Shilova A. Katrutsa Daniel Bershatsky ... Lionel Eyraud-Dubois Oleg Shlyazhko Denis Dimitrov Ivan Oseledets Olivier Beaumont 22 10 0 21 Feb 2022
ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training Jianfei Chen Lianmin Zheng Z. Yao Dequan Wang Ion Stoica Michael W. Mahoney Joseph E. Gonzalez MQ 27 74 0 29 Apr 2021
Optimal checkpointing for heterogeneous chains: how to train deep neural networks with limited memory Julien Herrmann Olivier Beaumont Lionel Eyraud-Dubois J. Herrmann Alexis Joly Alena Shilova BDL 28 29 0 27 Nov 2019
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks Chelsea Finn Pieter Abbeel Sergey Levine OOD 362 11,700 0 09 Mar 2017