Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation

17 May 2019

Papers citing "Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation"

50 / 50 papers shown

Title
Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniques Sanjay Surendranath Girija Shashank Kapoor Lakshit Arora Dipen Pradhan Aman Raj Ankit Shetgaonkar 106 0 0 05 May 2025
Learning Critically: Selective Self Distillation in Federated Learning on Non-IID Data Yuting He Yiqiang Chen Xiaodong Yang H. Yu Yi-Hua Huang Yang Gu FedML 143 21 0 20 Apr 2025
Earth-Adapter: Bridge the Geospatial Domain Gaps with Mixture of Frequency Adaptation Xiaoxing Hu Ziyang Gong Yansen Wang Yuru Jia Gen Luo Xue Yang 378 0 0 08 Apr 2025
A Lightweight and Extensible Cell Segmentation and Classification Model for Whole Slide Images N. Shvetsov T. Kilvaer M. Tafavvoghi Anders Sildnes Kajsa Møllersen Lill-ToveRasmussen Busund L. A. Bongo VLM 93 1 0 26 Feb 2025
CR-CTC: Consistency regularization on CTC for improved speech recognition Zengwei Yao Wei Kang Xiaoyu Yang Fangjun Kuang Liyong Guo Han Zhu Zengrui Jin Zhaoqing Li Long Lin Daniel Povey 107 4 0 17 Feb 2025
Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challenges Nayoung Lee Ziyang Cai Avi Schwarzschild Kangwook Lee Dimitris Papailiopoulos ReLM VLM LRM AI4CE 117 7 0 03 Feb 2025
sDREAMER: Self-distilled Mixture-of-Modality-Experts Transformer for Automatic Sleep Staging Jingyuan Chen Yuan Yao Mie Anderson Natalie Hauglund Celia Kjaerby Verena Untiet Maiken Nedergaard Jiebo Luo 113 2 0 28 Jan 2025
QCS: Feature Refining from Quadruplet Cross Similarity for Facial Expression Recognition Cong Wang Li Chen Lili Wang Zhaofan Li Xuebin Lv 123 1 0 28 Jan 2025
The Effect of Optimal Self-Distillation in Noisy Gaussian Mixture Model Kaito Takanami Takashi Takahashi Ayaka Sakata 82 1 0 27 Jan 2025
Hybrid Attention Model Using Feature Decomposition and Knowledge Distillation for Glucose Forecasting Ebrahim Farahmand Shovito Barua Soumma Nooshin Taheri Chatrudi Hassan Ghasemzadeh 61 2 0 16 Nov 2024
A Review of Pseudo-Labeling for Computer Vision Patrick Kage Jay C. Rothenberger Pavlos Andreadis Dimitrios I. Diochnos VLM 83 6 0 13 Aug 2024
Network Fission Ensembles for Low-Cost Self-Ensembles Hojung Lee Jong-Seok Lee UQCV 118 1 0 05 Aug 2024
DistilDoc: Knowledge Distillation for Visually-Rich Document Applications Jordy Van Landeghem Subhajit Maity Ayan Banerjee Matthew Blaschko Marie-Francine Moens Josep Lladós Sanket Biswas 96 2 0 12 Jun 2024
Improving Multi-task Learning via Seeking Task-based Flat Regions Hoang Phan Lam C. Tran Ngoc N. Tran Nhat Ho Tuan Truong Qi Lei Nhat Ho Dinh Q. Phung Trung Le 190 11 0 24 Nov 2022
HAPI: Hardware-Aware Progressive Inference Stefanos Laskaridis Stylianos I. Venieris Hyeji Kim Nicholas D. Lane 55 46 0 10 Aug 2020
Slimmable Neural Networks Jiahui Yu L. Yang N. Xu Jianchao Yang Thomas Huang 73 552 0 21 Dec 2018
MEAL: Multi-Model Ensemble via Adversarial Learning Zhiqiang Shen Zhankui He Xiangyang Xue AAML FedML 65 146 0 06 Dec 2018
KTAN: Knowledge Transfer Adversarial Network Peiye Liu Wu Liu Huadong Ma Tao Mei Mingoo Seok GAN 61 28 0 18 Oct 2018
Born Again Neural Networks Tommaso Furlanello Zachary Chase Lipton Michael Tschannen Laurent Itti Anima Anandkumar 68 1,030 0 12 May 2018
Label Refinery: Improving ImageNet Classification through Label Progression Hessam Bagherinezhad Maxwell Horton Mohammad Rastegari Ali Farhadi 59 190 0 07 May 2018
Stochastic Downsampling for Cost-Adjustable Inference and Improved Regularization in Convolutional Networks Jason Kuen Xiangfei Kong Zhe Lin G. Wang Jianxiong Yin Simon See Yap-Peng Tan BDL 54 25 0 29 Jan 2018
Convolutional Networks with Adaptive Inference Graphs Andreas Veit Serge J. Belongie OOD GNN 93 385 0 30 Nov 2017
SkipNet: Learning Dynamic Routing in Convolutional Networks Xin Wang Feng Yu Zi-Yi Dou Trevor Darrell Joseph E. Gonzalez 99 636 0 26 Nov 2017
BlockDrop: Dynamic Inference Paths in Residual Networks Zuxuan Wu Tushar Nagarajan Abhishek Kumar Steven J. Rennie L. Davis Kristen Grauman Rogerio Feris 87 466 0 22 Nov 2017
Focal Loss for Dense Object Detection Nayeon Lee Priya Goyal Ross B. Girshick Kaiming He Piotr Dollár ObjD 112 3,000 0 07 Aug 2017
Deep Mutual Learning Ying Zhang Tao Xiang Timothy M. Hospedales Huchuan Lu FedML 148 1,651 0 01 Jun 2017
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications Andrew G. Howard Menglong Zhu Bo Chen Dmitry Kalenichenko Weijun Wang Tobias Weyand M. Andreetto Hartwig Adam 3DH 1.1K 20,832 0 17 Apr 2017
Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer Sergey Zagoruyko N. Komodakis 116 2,579 0 12 Dec 2016
Feature Pyramid Networks for Object Detection Nayeon Lee Piotr Dollár Ross B. Girshick Kaiming He Bharath Hariharan Serge J. Belongie ObjD 461 22,102 0 09 Dec 2016
Aggregated Residual Transformations for Deep Neural Networks Saining Xie Ross B. Girshick Piotr Dollár Zhuowen Tu Kaiming He 509 10,318 0 16 Nov 2016
Deep Pyramidal Residual Networks Dongyoon Han Jiwhan Kim Junmo Kim 93 694 0 10 Oct 2016
Impatient DNNs - Deep Neural Networks with Dynamic Time Budgets Manuel Amthor E. Rodner Joachim Denzler 77 17 0 10 Oct 2016
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima N. Keskar Dheevatsa Mudigere J. Nocedal M. Smelyanskiy P. T. P. Tang ODL 421 2,936 0 15 Sep 2016
3D Deeply Supervised Network for Automatic Liver Segmentation from CT Volumes Qi Dou Hao Chen Yueming Jin Lequan Yu J. Qin Pheng-Ann Heng MedIm 76 345 0 03 Jul 2016
Wide Residual Networks Sergey Zagoruyko N. Komodakis 332 7,980 0 23 May 2016
Deep Networks with Stochastic Depth Gao Huang Yu Sun Zhuang Liu Daniel Sedra Kilian Q. Weinberger 209 2,356 0 30 Mar 2016
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks Mohammad Rastegari Vicente Ordonez Joseph Redmon Ali Farhadi MQ 161 4,356 0 16 Mar 2016
Deep Residual Learning for Image Recognition Kaiming He Xinming Zhang Shaoqing Ren Jian Sun MedIm 2.2K 193,814 0 10 Dec 2015
SSD: Single Shot MultiBox Detector Wen Liu Dragomir Anguelov D. Erhan Christian Szegedy Scott E. Reed Cheng-Yang Fu Alexander C. Berg ObjD BDL 224 29,808 0 08 Dec 2015
Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks Nicolas Papernot Patrick McDaniel Xi Wu S. Jha A. Swami AAML 92 3,072 0 14 Nov 2015
BinaryConnect: Training Deep Neural Networks with binary weights during propagations Matthieu Courbariaux Yoshua Bengio J. David MQ 206 2,985 0 02 Nov 2015
Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding Song Han Huizi Mao W. Dally 3DGS 253 8,833 0 01 Oct 2015
Cross Modal Distillation for Supervision Transfer Saurabh Gupta Judy Hoffman Jitendra Malik 104 536 0 02 Jul 2015
Learning both Weights and Connections for Efficient Neural Networks Song Han Jeff Pool J. Tran W. Dally CVBM 310 6,669 0 08 Jun 2015
Distilling the Knowledge in a Neural Network Geoffrey E. Hinton Oriol Vinyals J. Dean FedML 342 19,634 0 09 Mar 2015
FitNets: Hints for Thin Deep Nets Adriana Romero Nicolas Ballas Samira Ebrahimi Kahou Antoine Chassang C. Gatta Yoshua Bengio FedML 298 3,883 0 19 Dec 2014
Deeply-Supervised Nets Chen-Yu Lee Saining Xie Patrick W. Gallagher Zhengyou Zhang Zhuowen Tu 339 2,240 0 18 Sep 2014
Very Deep Convolutional Networks for Large-Scale Image Recognition Karen Simonyan Andrew Zisserman FAtt MDE 1.6K 100,330 0 04 Sep 2014
Neural Machine Translation by Jointly Learning to Align and Translate Dzmitry Bahdanau Kyunghyun Cho Yoshua Bengio AIMat 546 27,295 0 01 Sep 2014
Recurrent Models of Visual Attention Volodymyr Mnih N. Heess Alex Graves Koray Kavukcuoglu VLM 152 3,656 0 24 Jun 2014