v1v2 (latest)

AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networks

23 June 2021

Dan Alistarh

ArXiv (abs)PDF HTML Github (23★)

Papers citing "AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networks"

46 / 46 papers shown

Title
Dynamic Sparse Training versus Dense Training: The Unexpected Winner in Image Corruption Robustness Boqian Wu Q. Xiao Shunxin Wang N. Strisciuglio Mykola Pechenizkiy M. V. Keulen Decebal Constantin Mocanu Elena Mocanu OOD 3DH 198 3 0 03 Oct 2024
Mask in the Mirror: Implicit Sparsification Tom Jacobs R. Burkholz 182 4 0 19 Aug 2024
SequentialAttention++ for Block Sparsification: Differentiable Pruning Meets Combinatorial Optimization T. Yasuda Kyriakos Axiotis Gang Fu M. Bateni Vahab Mirrokni 173 0 0 27 Feb 2024
End-to-end Feature Selection Approach for Learning Skinny Trees Shibal Ibrahim Kayhan Behdin Rahul Mazumder 503 0 0 28 Oct 2023
Top-KAST: Top-K Always Sparse Training Siddhant M. Jayakumar Razvan Pascanu Jack W. Rae Simon Osindero Erich Elsen 157 100 0 07 Jun 2021
Accelerating Sparse Deep Neural Networks Asit K. Mishra J. Latorre Jeff Pool Darko Stosic Dusan Stosic Ganesh Venkatesh Chong Yu Paulius Micikevicius 160 235 0 16 Apr 2021
Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch Aojun Zhou Yukun Ma Junnan Zhu Jianbo Liu Zhijie Zhang Kun Yuan Wenxiu Sun Hongsheng Li 207 248 0 08 Feb 2021
Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks Torsten Hoefler Dan Alistarh Tal Ben-Nun Nikoli Dryden Alexandra Peste MQ 314 724 0 31 Jan 2021
TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems R. David Jared Duke Advait Jain Vijay Janapa Reddi Nat Jeffries ... Meghna Natraj Shlomi Regev Rocky Rhodes Tiezhen Wang Pete Warden 243 481 0 17 Oct 2020
Sparse Convex Optimization via Adaptively Regularized Hard Thresholding Kyriakos Axiotis M. Sviridenko 122 16 0 25 Jun 2020
Dynamic Model Pruning with Feedback Tao R. Lin Sebastian U. Stich Luis Barba Daniil Dmitriev Martin Jaggi 155 204 0 12 Jun 2020
Pruning neural networks without any data by iteratively conserving synaptic flow Hidenori Tanaka D. Kunin Daniel L. K. Yamins Surya Ganguli 176 648 0 09 Jun 2020
Soft Threshold Weight Reparameterization for Learnable Sparsity Aditya Kusupati Vivek Ramanujan Raghav Somani Mitchell Wortsman Prateek Jain Sham Kakade Ali Farhadi 148 247 0 08 Feb 2020
Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networks Aleksandr Shevchenko Marco Mondelli 169 37 0 20 Dec 2019
PyTorch: An Imperative Style, High-Performance Deep Learning Library Adam Paszke Sam Gross Francisco Massa Adam Lerer James Bradbury ... Sasank Chilamkurthy Benoit Steiner Lu Fang Junjie Bai Soumith Chintala ODL 544 42,591 0 03 Dec 2019
Rigging the Lottery: Making All Tickets Winners Utku Evci Trevor Gale Jacob Menick Pablo Samuel Castro Erich Elsen 197 607 0 25 Nov 2019
Fast Sparse ConvNets Erich Elsen Marat Dukhan Trevor Gale Karen Simonyan 172 153 0 21 Nov 2019
Understanding Top-k Sparsification in Distributed Deep Learning Shaoshuai Shi Xiaowen Chu Ka Chun Cheung Simon See 226 101 0 20 Nov 2019
Sparse Networks from Scratch: Faster Training without Losing Performance Tim Dettmers Luke Zettlemoyer 145 340 0 10 Jul 2019
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes Yang You Jing Li Sashank J. Reddi Jonathan Hseu Sanjiv Kumar Srinadh Bhojanapalli Xiaodan Song J. Demmel Kurt Keutzer Cho-Jui Hsieh ODL 265 999 0 01 Apr 2019
The State of Sparsity in Deep Neural Networks Trevor Gale Erich Elsen Sara Hooker 163 762 0 25 Feb 2019
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context Zihang Dai Zhilin Yang Yiming Yang J. Carbonell Quoc V. Le Ruslan Salakhutdinov VLM 260 3,745 0 09 Jan 2019
A Convergence Theory for Deep Learning via Over-Parameterization Zeyuan Allen-Zhu Yuanzhi Li Zhao Song AI4CE ODL 266 1,469 0 09 Nov 2018
SNIP: Single-shot Network Pruning based on Connection Sensitivity Namhoon Lee Thalaiyasingam Ajanthan Philip Torr VLM 269 1,207 0 04 Oct 2018
The Convergence of Sparsified Gradient Methods Dan Alistarh Torsten Hoefler M. Johansson Sarit Khirirat Nikola Konstantinov Cédric Renggli 169 494 0 27 Sep 2018
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks Jonathan Frankle Michael Carbin 263 3,485 0 09 Mar 2018
TVM: An Automated End-to-End Optimizing Compiler for Deep Learning Tianqi Chen T. Moreau Ziheng Jiang Lianmin Zheng Eddie Q. Yan ... Leyuan Wang Yuwei Hu Luis Ceze Carlos Guestrin Arvind Krishnamurthy 193 374 0 12 Feb 2018
Deep Rewiring: Training very sparse deep networks G. Bellec David Kappel Wolfgang Maass Robert Legenstein BDL 164 279 0 14 Nov 2017
To prune, or not to prune: exploring the efficacy of pruning for model compression Michael Zhu Suyog Gupta 197 1,281 0 05 Oct 2017
Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science Decebal Constantin Mocanu Elena Mocanu Peter Stone Phuong H. Nguyen M. Gibescu A. Liotta 178 634 0 15 Jul 2017
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 786 132,363 0 12 Jun 2017
Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon Xin Luna Dong Shangyu Chen Sinno Jialin Pan 183 506 0 22 May 2017
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications Andrew G. Howard Menglong Zhu Bo Chen Dmitry Kalenichenko Weijun Wang Tobias Weyand M. Andreetto Hartwig Adam 3DH 1.2K 20,892 0 17 Apr 2017
Variational Dropout Sparsifies Deep Neural Networks Dmitry Molchanov Arsenii Ashukha Dmitry Vetrov BDL 150 831 0 19 Jan 2017
Understanding deep learning requires rethinking generalization Chiyuan Zhang Samy Bengio Moritz Hardt Benjamin Recht Oriol Vinyals HAI 351 4,635 0 10 Nov 2016
Pointer Sentinel Mixture Models Stephen Merity Caiming Xiong James Bradbury R. Socher RALM 341 2,898 0 26 Sep 2016
Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition Hamed Karimi J. Nutini Mark Schmidt 280 1,221 0 16 Aug 2016
Training Skinny Deep Neural Networks with Iterative Hard Thresholding Methods Xiaojie Jin Xiao-Tong Yuan Jiashi Feng Shuicheng Yan 396 78 0 19 Jul 2016
Wide Residual Networks Sergey Zagoruyko N. Komodakis 353 8,000 0 23 May 2016
Binarized Neural Networks Itay Hubara Daniel Soudry Ran El-Yaniv MQ 204 1,348 0 08 Feb 2016
Deep Residual Learning for Image Recognition Kaiming He Xinming Zhang Shaoqing Ren Jian Sun MedIm 2.2K 194,426 0 10 Dec 2015
Learning both Weights and Connections for Efficient Neural Networks Song Han Jeff Pool J. Tran W. Dally CVBM 313 6,700 0 08 Jun 2015
Adam: A Method for Stochastic Optimization Diederik P. Kingma Jimmy Ba ODL 2.1K 150,312 0 22 Dec 2014
On Iterative Hard Thresholding Methods for High-dimensional M-Estimation Prateek Jain Ambuj Tewari Purushottam Kar 163 232 0 20 Oct 2014
ImageNet Large Scale Visual Recognition Challenge Olga Russakovsky Jia Deng Hao Su J. Krause S. Satheesh ... A. Karpathy A. Khosla Michael S. Bernstein Alexander C. Berg Li Fei-Fei VLM ObjD 1.7K 39,595 0 01 Sep 2014
Gradient Hard Thresholding Pursuit for Sparsity-Constrained Optimization Xiao-Tong Yuan Ping Li Tong Zhang 181 113 0 22 Nov 2013