EPSD: Early Pruning with Self-Distillation for Efficient Model Compression

31 January 2024

Papers citing "EPSD: Early Pruning with Self-Distillation for Efficient Model Compression"

50 / 50 papers shown

Title
CP $^3$ : Channel Pruning Plug-in for Point-based Networks Yaomin Huang Ning Liu Zhengping Che Zhiyuan Xu Yaxin Peng Chaomin Shen Guixu Zhang Xinmei Liu Feifei Feng Jian Tang 3DPC 71 14 0 23 Mar 2023
Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models Xiuying Wei Yunchen Zhang Xiangguo Zhang Ruihao Gong Shanghang Zhang Qi Zhang F. Yu Xianglong Liu MQ 92 152 0 27 Sep 2022
Training Your Sparse Neural Network Better with Any Mask Ajay Jaiswal Haoyu Ma Tianlong Chen Ying Ding Zhangyang Wang CVBM 108 36 0 26 Jun 2022
Prospect Pruning: Finding Trainable Weights at Initialization using Meta-Gradients Milad Alizadeh Shyam A. Tailor L. Zintgraf Joost R. van Amersfoort Sebastian Farquhar Nicholas D. Lane Y. Gal 125 41 0 16 Feb 2022
MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer Sachin Mehta Mohammad Rastegari ViT 285 1,274 0 05 Oct 2021
Prune Your Model Before Distill It Jinhyuk Park Albert No VLM 95 27 0 30 Sep 2021
Dynamical Isometry: The Missing Ingredient for Neural Network Pruning Huan Wang Can Qin Yue Bai Y. Fu 40 5 0 12 May 2021
Distilling Knowledge via Knowledge Review Pengguang Chen Shu Liu Hengshuang Zhao Jiaya Jia 189 442 0 19 Apr 2021
Recent Advances on Neural Network Pruning at Initialization Huan Wang Can Qin Yue Bai Yulun Zhang Yun Fu CVBM 73 67 0 11 Mar 2021
Unveiling the Potential of Structure Preserving for Weakly Supervised Object Localization Xingjia Pan Yingguo Gao Zhiwen Lin Fan Tang Weiming Dong Haolei Yuan Feiyue Huang Changsheng Xu WSOL 62 87 0 08 Mar 2021
Pruning Neural Networks at Initialization: Why are We Missing the Mark? Jonathan Frankle Gintare Karolina Dziugaite Daniel M. Roy Michael Carbin 65 240 0 18 Sep 2020
Single Shot Structured Pruning Before Training Joost R. van Amersfoort Milad Alizadeh Sebastian Farquhar Nicholas D. Lane Y. Gal 41 22 0 01 Jul 2020
Self-Knowledge Distillation with Progressive Refinement of Targets Kyungyul Kim Byeongmoon Ji Doyoung Yoon Sangheum Hwang ODL 76 182 0 22 Jun 2020
Progressive Skeletonization: Trimming more fat from a network at initialization Pau de Jorge Amartya Sanyal Harkirat Singh Behl Philip Torr Grégory Rogez P. Dokania 74 95 0 16 Jun 2020
Pruning neural networks without any data by iteratively conserving synaptic flow Hidenori Tanaka D. Kunin Daniel L. K. Yamins Surya Ganguli 169 648 0 09 Jun 2020
Regularizing Class-wise Predictions via Self-knowledge Distillation Sukmin Yun Jongjin Park Kimin Lee Jinwoo Shin 65 279 0 31 Mar 2020
Channel Pruning Guided by Classification Loss and Feature Importance Jinyang Guo Wanli Ouyang Dong Xu 51 54 0 15 Mar 2020
Comparing Rewinding and Fine-tuning in Neural Network Pruning Alex Renda Jonathan Frankle Michael Carbin 275 388 0 05 Mar 2020
Self-Distillation Amplifies Regularization in Hilbert Space H. Mobahi Mehrdad Farajtabar Peter L. Bartlett 69 235 0 13 Feb 2020
Linear Mode Connectivity and the Lottery Ticket Hypothesis Jonathan Frankle Gintare Karolina Dziugaite Daniel M. Roy Michael Carbin MoMe 156 619 0 11 Dec 2019
PyTorch: An Imperative Style, High-Performance Deep Learning Library Adam Paszke Sam Gross Francisco Massa Adam Lerer James Bradbury ... Sasank Chilamkurthy Benoit Steiner Lu Fang Junjie Bai Soumith Chintala ODL 520 42,559 0 03 Dec 2019
Rigging the Lottery: Making All Tickets Winners Utku Evci Trevor Gale Jacob Menick Pablo Samuel Castro Erich Elsen 197 602 0 25 Nov 2019
Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks Ruihao Gong Xianglong Liu Shenghu Jiang Tian-Hao Li Peng Hu Jiazhen Lin F. Yu Junjie Yan MQ 70 459 0 14 Aug 2019
AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Rates Ning Liu Xiaolong Ma Zhiyuan Xu Yanzhi Wang Jian Tang Jieping Ye 70 186 0 06 Jul 2019
A Signal Propagation Perspective for Pruning Neural Networks at Initialization Namhoon Lee Thalaiyasingam Ajanthan Stephen Gould Philip Torr AAML 64 155 0 14 Jun 2019
Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation Linfeng Zhang Jiebo Song Anni Gao Jingwei Chen Chenglong Bao Kaisheng Ma FedML 76 863 0 17 May 2019
Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization Hesham Mostafa Xin Wang 79 314 0 15 Feb 2019
Improved Knowledge Distillation via Teacher Assistant Seyed Iman Mirzadeh Mehrdad Farajtabar Ang Li Nir Levine Akihiro Matsukawa H. Ghasemzadeh 100 1,080 0 09 Feb 2019
Snapshot Distillation: Teacher-Student Optimization in One Generation Chenglin Yang Lingxi Xie Chi Su Alan Yuille 75 193 0 01 Dec 2018
SNIP: Single-shot Network Pruning based on Connection Sensitivity Namhoon Lee Thalaiyasingam Ajanthan Philip Torr VLM 263 1,206 0 04 Oct 2018
Bias-Reduced Uncertainty Estimation for Deep Neural Classifiers Yonatan Geifman Guy Uziel Ran El-Yaniv UQCV 57 140 0 21 May 2018
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks Jonathan Frankle Michael Carbin 242 3,484 0 09 Mar 2018
Stronger generalization bounds for deep nets via a compression approach Sanjeev Arora Rong Ge Behnam Neyshabur Yi Zhang MLT AI4CE 86 643 0 14 Feb 2018
MobileNetV2: Inverted Residuals and Linear Bottlenecks Mark Sandler Andrew G. Howard Menglong Zhu A. Zhmoginov Liang-Chieh Chen 186 19,316 0 13 Jan 2018
Visualizing the Loss Landscape of Neural Nets Hao Li Zheng Xu Gavin Taylor Christoph Studer Tom Goldstein 254 1,896 0 28 Dec 2017
Deep Rewiring: Training very sparse deep networks G. Bellec David Kappel Wolfgang Maass Robert Legenstein BDL 150 278 0 14 Nov 2017
Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science Decebal Constantin Mocanu Elena Mocanu Peter Stone Phuong H. Nguyen M. Gibescu A. Liotta 175 633 0 15 Jul 2017
Sharp Minima Can Generalize For Deep Nets Laurent Dinh Razvan Pascanu Samy Bengio Yoshua Bengio ODL 122 774 0 15 Mar 2017
Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations Itay Hubara Matthieu Courbariaux Daniel Soudry Ran El-Yaniv Yoshua Bengio MQ 149 1,867 0 22 Sep 2016
Wide Residual Networks Sergey Zagoruyko N. Komodakis 349 7,995 0 23 May 2016
Fully Convolutional Networks for Semantic Segmentation Evan Shelhamer Jonathan Long Trevor Darrell VOS SSeg 741 37,886 0 20 May 2016
Deep Residual Learning for Image Recognition Kaiming He Xinming Zhang Shaoqing Ren Jian Sun MedIm 2.2K 194,322 0 10 Dec 2015
Rethinking the Inception Architecture for Computer Vision Christian Szegedy Vincent Vanhoucke Sergey Ioffe Jonathon Shlens Z. Wojna 3DV BDL 886 27,412 0 02 Dec 2015
Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding Song Han Huizi Mao W. Dally 3DGS 263 8,854 0 01 Oct 2015
Learning both Weights and Connections for Efficient Neural Networks Song Han Jeff Pool J. Tran W. Dally CVBM 313 6,694 0 08 Jun 2015
Accelerating Very Deep Convolutional Networks for Classification and Detection Xinming Zhang Jianhua Zou Kaiming He Jian Sun 70 797 0 26 May 2015
Distilling the Knowledge in a Neural Network Geoffrey E. Hinton Oriol Vinyals J. Dean FedML 362 19,723 0 09 Mar 2015
FitNets: Hints for Thin Deep Nets Adriana Romero Nicolas Ballas Samira Ebrahimi Kahou Antoine Chassang C. Gatta Yoshua Bengio FedML 308 3,893 0 19 Dec 2014
Very Deep Convolutional Networks for Large-Scale Image Recognition Karen Simonyan Andrew Zisserman FAtt MDE 1.7K 100,479 0 04 Sep 2014
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks Andrew M. Saxe James L. McClelland Surya Ganguli ODL 181 1,849 0 20 Dec 2013