ActNN: Reducing Training Memory Footprint via 2-Bit Activation
Compressed Training

ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training

29 April 2021

Jianfei Chen

Michael W. Mahoney

Joseph E. Gonzalez

Papers citing "ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training"

16 / 16 papers shown

Title
PaCA: Partial Connection Adaptation for Efficient Fine-Tuning Sunghyeon Woo Sol Namkung Sunwoo Lee Inho Jeong Beomseok Kim Dongsuk Jeon 39 0 0 28 Feb 2025
COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Training Haocheng Xi Han Cai Ligeng Zhu Yaojie Lu Kurt Keutzer Jianfei Chen Song Han MQ 75 9 0 25 Oct 2024
Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation Yuchen Yang Yingdong Shi Cheems Wang Xiantong Zhen Yuxuan Shi Jun Xu 37 1 0 24 Jun 2024
Do We Really Need a Large Number of Visual Prompts? Youngeun Kim Yuhang Li Abhishek Moitra Ruokai Yin Priyadarshini Panda VLM VPVLM 40 5 0 26 May 2023
End-to-end codesign of Hessian-aware quantized neural networks for FPGAs and ASICs Javier Campos Zhen Dong Javier Mauricio Duarte A. Gholami Michael W. Mahoney Jovan Mitrevski Nhan Tran MQ 29 3 0 13 Apr 2023
An Evaluation of Memory Optimization Methods for Training Neural Networks Xiaoxuan Liu Siddharth Jha Alvin Cheung 26 0 0 26 Mar 2023
A Survey on Efficient Training of Transformers Bohan Zhuang Jing Liu Zizheng Pan Haoyu He Yuetian Weng Chunhua Shen 31 47 0 02 Feb 2023
From Node Interaction to Hop Interaction: New Effective and Scalable Graph Learning Paradigm Jie Chen Zilong Li Ying Zhu Junping Zhang Jian Pu 34 8 0 21 Nov 2022
OLLA: Optimizing the Lifetime and Location of Arrays to Reduce the Memory Usage of Neural Networks Benoit Steiner Mostafa Elhoushi Jacob Kahn James Hegarty 29 8 0 24 Oct 2022
A Comprehensive Study on Large-Scale Graph Training: Benchmarking and Rethinking Keyu Duan Zirui Liu Peihao Wang Wenqing Zheng Kaixiong Zhou Tianlong Chen Xia Hu Zhangyang Wang GNN 41 57 0 14 Oct 2022
POET: Training Neural Networks on Tiny Devices with Integrated Rematerialization and Paging Shishir G. Patil Paras Jain P. Dutta Ion Stoica Joseph E. Gonzalez 12 35 0 15 Jul 2022
DropIT: Dropping Intermediate Tensors for Memory-Efficient DNN Training Joya Chen Kai Xu Yuhui Wang Yifei Cheng Angela Yao 19 7 0 28 Feb 2022
Mesa: A Memory-saving Training Framework for Transformers Zizheng Pan Peng Chen Haoyu He Jing Liu Jianfei Cai Bohan Zhuang 25 20 0 22 Nov 2021
ZeRO-Offload: Democratizing Billion-Scale Model Training Jie Ren Samyam Rajbhandari Reza Yazdani Aminabadi Olatunji Ruwase Shuangyang Yang Minjia Zhang Dong Li Yuxiong He MoE 177 414 0 18 Jan 2021
Improved Baselines with Momentum Contrastive Learning Xinlei Chen Haoqi Fan Ross B. Girshick Kaiming He SSL 267 3,371 0 09 Mar 2020
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 245 1,821 0 17 Sep 2019