Exploring Dark Knowledge under Various Teacher Capacities and Addressing Capacity Mismatch

21 May 2024

Papers citing "Exploring Dark Knowledge under Various Teacher Capacities and Addressing Capacity Mismatch"

49 / 49 papers shown

Title
Logit Standardization in Knowledge Distillation Shangquan Sun Wenqi Ren Jingzhi Li Rui Wang Xiaochun Cao 77 69 0 03 Mar 2024
All Roads Lead to Rome? Exploring the Invariance of Transformers' Representations Yuxin Ren Qipeng Guo Zhijing Jin Shauli Ravfogel Mrinmaya Sachan Bernhard Schölkopf Ryan Cotterell 55 4 0 23 May 2023
Efficient Knowledge Distillation from Model Checkpoints Chaofei Wang Qisen Yang Rui Huang S. Song Gao Huang FedML 39 35 0 12 Oct 2022
Asymmetric Temperature Scaling Makes Larger Networks Teach Well Again Xin-Chun Li Wenxuan Fan Shaoming Song Yinchuan Li Bingshuai Li Yunfeng Shao De-Chuan Zhan 73 30 0 10 Oct 2022
Revisiting Label Smoothing and Knowledge Distillation Compatibility: What was Missing? Keshigeyan Chandrasegaran Ngoc-Trung Tran Yunqing Zhao Ngai-Man Cheung 120 44 0 29 Jun 2022
Knowledge Distillation from A Stronger Teacher Tao Huang Shan You Fei Wang Chao Qian Chang Xu 65 247 0 21 May 2022
On Representation Knowledge Distillation for Graph Neural Networks Chaitanya K. Joshi Fayao Liu Xu Xun Jie Lin Chuan-Sheng Foo 67 60 0 09 Nov 2021
Few-Shot Learning with a Strong Teacher Han-Jia Ye Lu Ming De-Chuan Zhan Wei-Lun Chao 68 52 0 01 Jul 2021
Towards Understanding Knowledge Distillation Mary Phuong Christoph H. Lampert 63 318 0 27 May 2021
Knowledge Distillation as Semiparametric Inference Tri Dao G. Kamath Vasilis Syrgkanis Lester W. Mackey 66 31 0 20 Apr 2021
Generalization bounds via distillation Daniel J. Hsu Ziwei Ji Matus Telgarsky Lan Wang FedML 46 34 0 12 Apr 2021
Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study Zhiqiang Shen Zechun Liu Dejia Xu Zitian Chen Kwang-Ting Cheng Marios Savvides 41 76 0 01 Apr 2021
Distilling a Powerful Student Model via Online Knowledge Distillation Shaojie Li Mingbao Lin Yan Wang Yongjian Wu Yonghong Tian Ling Shao Rongrong Ji FedML 72 47 0 26 Mar 2021
LogME: Practical Assessment of Pre-trained Models for Transfer Learning Kaichao You Yong Liu Jianmin Wang Mingsheng Long 79 183 0 22 Feb 2021
Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective Helong Zhou Liangchen Song Jiajie Chen Ye Zhou Guoli Wang Junsong Yuan Qian Zhang 67 174 0 01 Feb 2021
Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher Guangda Ji Zhanxing Zhu 73 44 0 20 Oct 2020
Embedded Encoder-Decoder in Convolutional Networks Towards Explainable AI A. Tavanaei XAI 84 31 0 19 Jun 2020
Knowledge Distillation: A Survey Jianping Gou B. Yu Stephen J. Maybank Dacheng Tao VLM 64 2,946 0 09 Jun 2020
ResKD: Residual-Guided Knowledge Distillation Xuewei Li Songyuan Li Bourahla Omar Leilei Gan Xi Li 47 48 0 08 Jun 2020
Explaining Knowledge Distillation by Quantifying the Knowledge Xu Cheng Zhefan Rao Yilan Chen Quanshi Zhang 53 123 0 07 Mar 2020
Understanding and Improving Knowledge Distillation Jiaxi Tang Rakesh Shivanna Zhe Zhao Dong Lin Anima Singh Ed H. Chi Sagar Jain 63 131 0 10 Feb 2020
Knowledge Representing: Efficient, Sparse Representation of Prior Knowledge for Knowledge Distillation Junjie Liu Dongchao Wen Hongxing Gao Wei Tao Tse-Wei Chen Kinya Osa Masami Kato 50 21 0 13 Nov 2019
Contrastive Representation Distillation Yonglong Tian Dilip Krishnan Phillip Isola 144 1,048 0 23 Oct 2019
On the Efficacy of Knowledge Distillation Ligang He Rui Mao 92 609 0 03 Oct 2019
Similarity-Preserving Knowledge Distillation Frederick Tung Greg Mori 113 976 0 23 Jul 2019
When Does Label Smoothing Help? Rafael Müller Simon Kornblith Geoffrey E. Hinton UQCV 189 1,943 0 06 Jun 2019
Learning Metrics from Teachers: Compact Networks for Image Embedding Lu Yu V. O. Yazici Xialei Liu Joost van de Weijer Yongmei Cheng Arnau Ramisa 48 108 0 07 Apr 2019
Correlation Congruence for Knowledge Distillation Baoyun Peng Xiao Jin Jiaheng Liu Shunfeng Zhou Yichao Wu Yu Liu Dongsheng Li Zhaoning Zhang 86 511 0 03 Apr 2019
A Principled Approach for Learning Task Similarity in Multitask Learning Changjian Shui Mahdieh Abbasi Louis-Émile Robitaille Boyu Wang Christian Gagné 62 57 0 21 Mar 2019
Improved Knowledge Distillation via Teacher Assistant Seyed Iman Mirzadeh Mehrdad Farajtabar Ang Li Nir Levine Akihiro Matsukawa H. Ghasemzadeh 92 1,074 0 09 Feb 2019
Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons Byeongho Heo Minsik Lee Sangdoo Yun J. Choi 55 525 0 08 Nov 2018
ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design Ningning Ma Xiangyu Zhang Haitao Zheng Jian Sun 174 4,983 0 30 Jul 2018
Born Again Neural Networks Tommaso Furlanello Zachary Chase Lipton Michael Tschannen Laurent Itti Anima Anandkumar 68 1,030 0 12 May 2018
Learning Deep Representations with Probabilistic Knowledge Transfer Nikolaos Passalis Anastasios Tefas 57 411 0 28 Mar 2018
Explicit Inductive Bias for Transfer Learning with Convolutional Networks Xuhong Li Yves Grandvalet Franck Davoine SSL 74 354 0 05 Feb 2018
MobileNetV2: Inverted Residuals and Linear Bottlenecks Mark Sandler Andrew G. Howard Menglong Zhu A. Zhmoginov Liang-Chieh Chen 178 19,271 0 13 Jan 2018
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices Xiangyu Zhang Xinyu Zhou Mengxiao Lin Jian Sun AI4TS 136 6,865 0 04 Jul 2017
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications Andrew G. Howard Menglong Zhu Bo Chen Dmitry Kalenichenko Weijun Wang Tobias Weyand M. Andreetto Hartwig Adam 3DH 1.1K 20,832 0 17 Apr 2017
Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer Sergey Zagoruyko N. Komodakis 116 2,578 0 12 Dec 2016
Aggregated Residual Transformations for Deep Neural Networks Saining Xie Ross B. Girshick Piotr Dollár Zhuowen Tu Kaiming He 509 10,318 0 16 Nov 2016
Wide Residual Networks Sergey Zagoruyko N. Komodakis 332 7,980 0 23 May 2016
Deep Residual Learning for Image Recognition Kaiming He Xinming Zhang Shaoqing Ren Jian Sun MedIm 2.2K 193,814 0 10 Dec 2015
Convergent Learning: Do different neural networks learn the same representations? Yixuan Li J. Yosinski Jeff Clune Hod Lipson John E. Hopcroft SSL 83 368 0 24 Nov 2015
Unifying distillation and privileged information David Lopez-Paz Léon Bottou Bernhard Schölkopf V. Vapnik FedML 165 462 0 11 Nov 2015
Distilling the Knowledge in a Neural Network Geoffrey E. Hinton Oriol Vinyals J. Dean FedML 342 19,634 0 09 Mar 2015
FitNets: Hints for Thin Deep Nets Adriana Romero Nicolas Ballas Samira Ebrahimi Kahou Antoine Chassang C. Gatta Yoshua Bengio FedML 298 3,883 0 19 Dec 2014
Unsupervised Domain Adaptation by Backpropagation Yaroslav Ganin Victor Lempitsky OOD 233 6,022 0 26 Sep 2014
Very Deep Convolutional Networks for Large-Scale Image Recognition Karen Simonyan Andrew Zisserman FAtt MDE 1.6K 100,330 0 04 Sep 2014
Barnes-Hut-SNE Laurens van der Maaten 80 211 0 15 Jan 2013