Layer Freezing & Data Sieving: Missing Pieces of a Generic Framework for Sparse Training

22 September 2022

Papers citing "Layer Freezing & Data Sieving: Missing Pieces of a Generic Framework for Sparse Training"

50 / 53 papers shown

Title
RoRA: Efficient Fine-Tuning of LLM with Reliability Optimization for Rank Adaptation Jun Liu Zhenglun Kong Peiyan Dong Changdi Yang Xuan Shen ... Wei Niu Wenbin Zhang Xue Lin Dong Huang Yanzhi Wang ALM 71 2 0 08 Jan 2025
Budgeted Online Continual Learning by Adaptive Layer Freezing and Frequency-based Sampling Minhyuk Seo Hyunseo Koh Jonghyun Choi 69 1 0 19 Oct 2024
Egeria: Efficient DNN Training with Knowledge-Guided Layer Freezing Yiding Wang D. Sun Kai Chen Fan Lai Mosharaf Chowdhury 80 44 0 17 Jan 2022
MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge Geng Yuan Xiaolong Ma Wei Niu Zhengang Li Zhenglun Kong ... Minghai Qin Bin Ren Yanzhi Wang Sijia Liu Xue Lin 43 92 0 26 Oct 2021
GRIM: A General, Real-Time Deep Learning Inference Framework for Mobile Devices based on Fine-Grained Structured Weight Sparsity Wei Niu Zhengang Xiaolong Ma Peiyan Dong Gang Zhou Xuehai Qian Xue Lin Yanzhi Wang Bin Ren 26 19 0 25 Aug 2021
Teachers Do More Than Teach: Compressing Image-to-Image Models Qing Jin Jian Ren Oliver J. Woodford Jiazhuo Wang Geng Yuan Yanzhi Wang Sergey Tulyakov 50 54 0 05 Mar 2021
Lottery Ticket Preserves Weight Correlation: Is It Desirable or Not? Ning Liu Geng Yuan Zhengping Che Xuan Shen Xiaolong Ma Qing Jin Jian Ren Jian Tang Sijia Liu Yanzhi Wang 55 31 0 19 Feb 2021
PipeTransformer: Automated Elastic Pipelining for Distributed Training of Transformers Chaoyang He Shen Li Mahdi Soltanolkotabi Salman Avestimehr 20 29 0 05 Feb 2021
AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning Yuhan Liu Saurabh Agarwal Shivaram Venkataraman OffRL 34 54 0 02 Feb 2021
Reservoir Transformers Sheng Shen Alexei Baevski Ari S. Morcos Kurt Keutzer Michael Auli Douwe Kiela 53 17 0 30 Dec 2020
Weight Update Skipping: Reducing Training Time for Artificial Neural Networks Pooneh Safayenikoo Ismail Akturk 9 15 0 05 Dec 2020
FreezeNet: Full Performance by Reduced Storage Costs Paul Wimmer Jens Mehnert Alexandru Paul Condurache 47 13 0 28 Nov 2020
Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping Minjia Zhang Yuxiong He AI4CE 26 101 0 26 Oct 2020
On the Transformer Growth for Progressive BERT Training Xiaotao Gu Liyuan Liu Hongkun Yu Jing Li Chong Chen Jiawei Han VLM 77 51 0 23 Oct 2020
Why Layer-Wise Learning is Hard to Scale-up and a Possible Solution via Accelerated Downsampling Wenchi Ma Miao Yu Kaidong Li Guanghui Wang 35 5 0 15 Oct 2020
Single Shot Structured Pruning Before Training Joost R. van Amersfoort Milad Alizadeh Sebastian Farquhar Nicholas D. Lane Y. Gal 34 22 0 01 Jul 2020
Pruning neural networks without any data by iteratively conserving synaptic flow Hidenori Tanaka D. Kunin Daniel L. K. Yamins Surya Ganguli 94 636 0 09 Jun 2020
RTMobile: Beyond Real-Time Mobile Acceleration of RNNs for Speech Recognition Peiyan Dong Siyue Wang Wei Niu Chengming Zhang Sheng Lin ... Yifan Gong Bin Ren Xinyu Lin Yanzhi Wang Dingwen Tao 25 45 0 19 Feb 2020
An Image Enhancing Pattern-based Sparsity for Real-time Inference on Mobile Devices Xiaolong Ma Wei Niu Tianyun Zhang Sijia Liu Sheng Lin ... Xiang Chen Jian Tang Kaisheng Ma Bin Ren Yanzhi Wang 52 27 0 20 Jan 2020
PatDNN: Achieving Real-Time DNN Execution on Mobile Devices with Pattern-based Weight Pruning Wei Niu Xiaolong Ma Sheng Lin Shihao Wang Xuehai Qian Xinyu Lin Yanzhi Wang Bin Ren MQ 50 227 0 01 Jan 2020
PyTorch: An Imperative Style, High-Performance Deep Learning Library Adam Paszke Sam Gross Francisco Massa Adam Lerer James Bradbury ... Sasank Chilamkurthy Benoit Steiner Lu Fang Junjie Bai Soumith Chintala ODL 209 42,038 0 03 Dec 2019
Rigging the Lottery: Making All Tickets Winners Utku Evci Trevor Gale Jacob Menick Pablo Samuel Castro Erich Elsen 116 592 0 25 Nov 2019
What Would Elsa Do? Freezing Layers During Transformer Fine-Tuning Jaejun Lee Raphael Tang Jimmy J. Lin 49 121 0 08 Nov 2019
Knowledge Distillation from Internal Representations Gustavo Aguilar Yuan Ling Yu Zhang Benjamin Yao Xing Fan Edward Guo 52 179 0 08 Oct 2019
PCONV: The Missing but Desirable Sparsity in DNN Weight Pruning for Real-time Execution on Mobile Devices Xiaolong Ma Fu-Ming Guo Wei Niu Xue Lin Jian Tang Kaisheng Ma Bin Ren Yanzhi Wang CVBM 40 176 0 06 Sep 2019
Sparse Networks from Scratch: Faster Training without Losing Performance Tim Dettmers Luke Zettlemoyer 74 337 0 10 Jul 2019
Non-Structured DNN Weight Pruning -- Is It Beneficial in Any Platform? Xiaolong Ma Sheng Lin Shaokai Ye Zhezhi He Linfeng Zhang ... Deliang Fan Xuehai Qian Xinyu Lin Kaisheng Ma Yanzhi Wang MQ 53 92 0 03 Jul 2019
Network Pruning via Transformable Architecture Search Xuanyi Dong Yi Yang 3DPC 44 241 0 23 May 2019
Similarity of Neural Network Representations Revisited Simon Kornblith Mohammad Norouzi Honglak Lee Geoffrey E. Hinton 120 1,382 0 01 May 2019
Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization Hesham Mostafa Xin Wang 57 309 0 15 Feb 2019
PruneTrain: Fast Neural Network Training by Dynamic Sparse Model Reconfiguration Sangkug Lym Esha Choukse Siavash Zangeneh W. Wen Sujay Sanghavi M. Erez CVBM 19 88 0 26 Jan 2019
An Empirical Study of Example Forgetting during Deep Neural Network Learning Mariya Toneva Alessandro Sordoni Rémi Tachet des Combes Adam Trischler Yoshua Bengio Geoffrey J. Gordon 95 723 0 12 Dec 2018
Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration Yang He Ping Liu Ziwei Wang Zhilan Hu Yi Yang AAML 3DPC 61 1,044 0 01 Nov 2018
SNIP: Single-shot Network Pruning based on Connection Sensitivity Namhoon Lee Thalaiyasingam Ajanthan Philip Torr VLM 192 1,190 0 04 Oct 2018
Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks Yang He Guoliang Kang Xuanyi Dong Yanwei Fu Yi Yang AAML VLM 45 960 0 21 Aug 2018
Efficient Hardware Realization of Convolutional Neural Networks using Intra-Kernel Regular Pruning Maurice Yang Mahmoud Faraj Assem Hussein V. Gaudet CVBM 44 12 0 15 Mar 2018
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks Jonathan Frankle Michael Carbin 157 3,433 0 09 Mar 2018
Compressing Neural Networks using the Variational Information Bottleneck Bin Dai Chen Zhu David Wipf MLT 47 180 0 28 Feb 2018
NISP: Pruning Networks using Neuron Importance Score Propagation Ruichi Yu Ang Li Chun-Fu Chen Jui-Hsin Lai Vlad I. Morariu Xintong Han M. Gao Ching-Yung Lin L. Davis 60 798 0 16 Nov 2017
Deep Rewiring: Training very sparse deep networks G. Bellec David Kappel Wolfgang Maass Robert Legenstein BDL 88 276 0 14 Nov 2017
ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression Jian-Hao Luo Jianxin Wu Weiyao Lin 35 1,751 0 20 Jul 2017
Channel Pruning for Accelerating Very Deep Neural Networks Yihui He Xiangyu Zhang Jian Sun 189 2,513 0 19 Jul 2017
Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science Decebal Constantin Mocanu Elena Mocanu Peter Stone Phuong H. Nguyen M. Gibescu A. Liotta 98 619 0 15 Jul 2017
FreezeOut: Accelerate Training by Progressively Freezing Layers Andrew Brock Theodore Lim J. Ritchie Nick Weston 32 123 0 15 Jun 2017
Structured Bayesian Pruning via Log-Normal Multiplicative Noise Kirill Neklyudov Dmitry Molchanov Arsenii Ashukha Dmitry Vetrov BDL 81 188 0 20 May 2017
Learning What Data to Learn Yang Fan Fei Tian Tao Qin Jiang Bian Tie-Yan Liu 34 79 0 28 Feb 2017
Variational Dropout Sparsifies Deep Neural Networks Dmitry Molchanov Arsenii Ashukha Dmitry Vetrov BDL 83 825 0 19 Jan 2017
Dynamic Network Surgery for Efficient DNNs Yiwen Guo Anbang Yao Yurong Chen 59 1,054 0 16 Aug 2016
Learning Structured Sparsity in Deep Neural Networks W. Wen Chunpeng Wu Yandan Wang Yiran Chen Hai Helen Li 102 2,331 0 12 Aug 2016
Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures Hengyuan Hu Rui Peng Yu-Wing Tai Chi-Keung Tang 45 885 0 12 Jul 2016