Teacher-Student Architecture for Knowledge Distillation: A Survey

8 August 2023

Xue Liu

Papers citing "Teacher-Student Architecture for Knowledge Distillation: A Survey"

50 / 142 papers shown

Title
Distilling Knowledge from Graph Convolutional Networks Yiding Yang Jiayan Qiu Xiuming Zhang Dacheng Tao Xinchao Wang 228 233 0 23 Mar 2020
Self-trained Deep Ordinal Regression for End-to-End Video Anomaly Detection Guansong Pang Cheng Yan Chunhua Shen Anton Van Den Hengel Xiao Bai 61 209 0 15 Mar 2020
Explaining Knowledge Distillation by Quantifying the Knowledge Xu Cheng Zhefan Rao Yilan Chen Quanshi Zhang 62 122 0 07 Mar 2020
An Efficient Method of Training Small Models for Regression Problems with Knowledge Distillation M. Takamoto Yusuke Morishita Hitoshi Imaoka 34 33 0 28 Feb 2020
Feature-map-level Online Adversarial Knowledge Distillation Inseop Chung Seonguk Park Jangho Kim Nojun Kwak GAN 81 131 0 05 Feb 2020
Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer Hong Chang Virat Shejwalkar Reza Shokri Amir Houmansadr FedML 83 169 0 24 Dec 2019
TextNAS: A Neural Architecture Search Space tailored for Text Representation Yujing Wang Yaming Yang Yiren Chen Jing Bai Ce Zhang Guinan Su Xiaoyu Kou Yunhai Tong Mao Yang Lidong Zhou 73 57 0 23 Dec 2019
Explaining Sequence-Level Knowledge Distillation as Data-Augmentation for Neural Machine Translation Mitchell A. Gordon Kevin Duh 41 28 0 06 Dec 2019
Online Knowledge Distillation with Diverse Peers Defang Chen Jian-Ping Mei Can Wang Yan Feng Chun-Yen Chen FedML 77 302 0 01 Dec 2019
Towards Oracle Knowledge Distillation with Neural Architecture Search Minsoo Kang Jonghwan Mun Bohyung Han FedML 95 44 0 29 Nov 2019
Few Shot Network Compression via Cross Distillation Haoli Bai Jiaxiang Wu Irwin King Michael Lyu FedML 70 60 0 21 Nov 2019
Collaborative Distillation for Top-N Recommendation Jae-woong Lee Minjin Choi Jongwuk Lee Hyunjung Shim 65 50 0 13 Nov 2019
Self-training with Noisy Student improves ImageNet classification Qizhe Xie Minh-Thang Luong Eduard H. Hovy Quoc V. Le NoLa 315 2,392 0 11 Nov 2019
Deep geometric knowledge distillation with graphs Carlos Lassance Myriam Bontonou G. B. Hacene Vincent Gripon Jian Tang Antonio Ortega 41 39 0 08 Nov 2019
Uninformed Students: Student-Teacher Anomaly Detection with Discriminative Latent Embeddings Paul Bergmann Michael Fauser David Sattlegger C. Steger 84 665 0 06 Nov 2019
Contrastive Representation Distillation Yonglong Tian Dilip Krishnan Phillip Isola 165 1,053 0 23 Oct 2019
Model Compression with Two-stage Multi-teacher Knowledge Distillation for Web Question Answering System Ze Yang Linjun Shou Ming Gong Wutao Lin Daxin Jiang 60 94 0 18 Oct 2019
VarGFaceNet: An Efficient Variable Group Convolutional Neural Network for Lightweight Face Recognition Mengjia Yan Mengao Zhao Zining Xu Qian Zhang Guoli Wang Zhizhong Su CVBM 69 92 0 11 Oct 2019
Knowledge Distillation from Internal Representations Gustavo Aguilar Yuan Ling Yu Zhang Benjamin Yao Xing Fan Edward Guo 80 181 0 08 Oct 2019
FedMD: Heterogenous Federated Learning via Model Distillation Daliang Li Junpu Wang FedML 101 862 0 08 Oct 2019
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter Victor Sanh Lysandre Debut Julien Chaumond Thomas Wolf 255 7,547 0 02 Oct 2019
TinyBERT: Distilling BERT for Natural Language Understanding Xiaoqi Jiao Yichun Yin Lifeng Shang Xin Jiang Xiao Chen Linlin Li F. Wang Qun Liu VLM 113 1,869 0 23 Sep 2019
Patient Knowledge Distillation for BERT Model Compression S. Sun Yu Cheng Zhe Gan Jingjing Liu 142 843 0 25 Aug 2019
Dynamic Kernel Distillation for Efficient Pose Estimation in Videos Xuecheng Nie Yuncheng Li Linjie Luo Ning Zhang Jiashi Feng 3DH 61 65 0 24 Aug 2019
Self-Knowledge Distillation in Natural Language Processing Sangchul Hahn Heeyoul Choi 82 112 0 02 Aug 2019
Distilling Knowledge From a Deep Pose Regressor Network Muhamad Risqi U. Saputra Pedro Porto Buarque de Gusmão Yasin Almalioglu Andrew Markham A. Trigoni 81 103 0 02 Aug 2019
Learn Spelling from Teachers: Transferring Knowledge from Language Models to Sequence-to-Sequence Speech Recognition Ye Bai Jiangyan Yi J. Tao Zhengkun Tian Zhengqi Wen KELM 66 38 0 13 Jul 2019
Compression of Acoustic Event Detection Models With Quantized Distillation Bowen Shi Ming Sun Chieh-Chi Kao Viktor Rozgic Spyros Matsoukas Chao Wang 53 14 0 01 Jul 2019
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks Mingxing Tan Quoc V. Le 3DV MedIm 156 18,179 0 28 May 2019
ShrinkTeaNet: Million-scale Lightweight Face Recognition via Shrinking Teacher-Student Networks C. Duong Khoa Luu Kha Gia Quach Ngan Le CVBM 55 39 0 25 May 2019
Adversarially Robust Distillation Micah Goldblum Liam H. Fowl Soheil Feizi Tom Goldstein AAML 65 210 0 23 May 2019
Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation Linfeng Zhang Jiebo Song Anni Gao Jingwei Chen Chenglong Bao Kaisheng Ma FedML 76 865 0 17 May 2019
Semi-supervised Acoustic Event Detection based on tri-training Bowen Shi Ming Sun Chieh-Chi Kao Viktor Rozgic Spyros Matsoukas Chao Wang 63 19 0 29 Apr 2019
Conditional Teacher-Student Learning Zhong Meng Jinyu Li Yong Zhao Jiawei Liu 60 91 0 28 Apr 2019
Bidirectional Learning for Domain Adaptation of Semantic Segmentation Yunsheng Li Lu Yuan Nuno Vasconcelos SSeg 95 629 0 24 Apr 2019
Feature Fusion for Online Mutual Knowledge Distillation Jangho Kim Minsung Hyun Inseop Chung Nojun Kwak FedML 65 91 0 19 Apr 2019
Audio-Visual Model Distillation Using Acoustic Images Andrés F. Pérez Valentina Sanguineti Pietro Morerio Vittorio Murino VLM 56 27 0 16 Apr 2019
Variational Information Distillation for Knowledge Transfer SungSoo Ahn S. Hu Andreas C. Damianou Neil D. Lawrence Zhenwen Dai 89 621 0 11 Apr 2019
Relational Knowledge Distillation Wonpyo Park Dongju Kim Yan Lu Minsu Cho 74 1,423 0 10 Apr 2019
White-to-Black: Efficient Distillation of Black-Box Adversarial Attacks Yotam Gil Yoav Chai O. Gorodissky Jonathan Berant MLAU AAML 48 46 0 04 Apr 2019
Correlation Congruence for Knowledge Distillation Baoyun Peng Xiao Jin Jiaheng Liu Shunfeng Zhou Yichao Wu Yu Liu Dongsheng Li Zhaoning Zhang 94 513 0 03 Apr 2019
M2KD: Multi-model and Multi-level Knowledge Distillation for Incremental Learning Peng Zhou Long Mai Jianming Zhang N. Xu Zuxuan Wu L. Davis CLL VLM 60 55 0 03 Apr 2019
Distilling Task-Specific Knowledge from BERT into Simple Neural Networks Raphael Tang Yao Lu Linqing Liu Lili Mou Olga Vechtomova Jimmy J. Lin 75 421 0 28 Mar 2019
Mixture Models for Diverse Machine Translation: Tricks of the Trade T. Shen Myle Ott Michael Auli MarcÁurelio Ranzato MoE 93 151 0 20 Feb 2019
Improved Knowledge Distillation via Teacher Assistant Seyed Iman Mirzadeh Mehrdad Farajtabar Ang Li Nir Levine Akihiro Matsukawa H. Ghasemzadeh 100 1,081 0 09 Feb 2019
Knowledge Distillation with Feature Maps for Image Classification Wei-Chun Chen Chia-Che Chang Chien-Yu Lu Che-Rung Lee 53 36 0 03 Dec 2018
Fast Human Pose Estimation Feng Zhang Xiatian Zhu Mao Ye 3DH 74 238 0 13 Nov 2018
Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons Byeongho Heo Minsik Lee Sangdoo Yun J. Choi 55 527 0 08 Nov 2018
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 1.8K 95,175 0 11 Oct 2018
Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender System Jiaxi Tang Ke Wang 64 189 0 19 Sep 2018