Understanding the Effects of Projectors in Knowledge Distillation

26 October 2023

Papers citing "Understanding the Effects of Projectors in Knowledge Distillation"

42 / 42 papers shown

Title
Efficient Knowledge Distillation from Model Checkpoints Chaofei Wang Qisen Yang Rui Huang S. Song Gao Huang FedML 46 36 0 12 Oct 2022
Masked Generative Distillation Zhendong Yang Zhe Li Mingqi Shao Dachuan Shi Zehuan Yuan Chun Yuan FedML 76 176 0 03 May 2022
Decoupled Knowledge Distillation Borui Zhao Quan Cui Renjie Song Yiyu Qiu Jiajun Liang 74 544 0 16 Mar 2022
A ConvNet for the 2020s Zhuang Liu Hanzi Mao Chaozheng Wu Christoph Feichtenhofer Trevor Darrell Saining Xie ViT 171 5,192 0 10 Jan 2022
A Fast Knowledge Distillation Framework for Visual Recognition Zhiqiang Shen Eric P. Xing VLM 63 48 0 02 Dec 2021
Revisiting the Calibration of Modern Neural Networks Matthias Minderer Josip Djolonga Rob Romijnders F. Hubis Xiaohua Zhai N. Houlsby Dustin Tran Mario Lucic UQCV 100 367 0 15 Jun 2021
Knowledge distillation: A good teacher is patient and consistent Lucas Beyer Xiaohua Zhai Amelie Royer L. Markeeva Rohan Anil Alexander Kolesnikov VLM 107 296 0 09 Jun 2021
Distilling Knowledge via Knowledge Review Pengguang Chen Shu Liu Hengshuang Zhao Jiaya Jia 189 442 0 19 Apr 2021
Show, Attend and Distill:Knowledge Distillation via Attention-based Feature Matching Mingi Ji Byeongho Heo Sungrae Park 110 148 0 05 Feb 2021
Training data-efficient image transformers & distillation through attention Hugo Touvron Matthieu Cord Matthijs Douze Francisco Massa Alexandre Sablayrolles Hervé Jégou ViT 389 6,793 0 23 Dec 2020
Cross-Layer Distillation with Semantic Calibration Defang Chen Jian-Ping Mei Yuan Zhang Can Wang Yan Feng Chun-Yen Chen FedML 88 298 0 06 Dec 2020
Exploring Simple Siamese Representation Learning Xinlei Chen Kaiming He SSL 258 4,067 0 20 Nov 2020
Interventional Few-Shot Learning Zhongqi Yue Hanwang Zhang Qianru Sun Xiansheng Hua 91 230 0 28 Sep 2020
Causal Intervention for Weakly-Supervised Semantic Segmentation Dong Zhang Hanwang Zhang Jinhui Tang Xiansheng Hua Qianru Sun CML ISeg 103 450 0 26 Sep 2020
Densely Guided Knowledge Distillation using Multiple Teacher Assistants Wonchul Son Jaemin Na Junyong Choi Wonjun Hwang 67 116 0 18 Sep 2020
MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without Tricks Zhiqiang Shen Marios Savvides 59 63 0 17 Sep 2020
Prime-Aware Adaptive Distillation Youcai Zhang Zhonghao Lan Yuchen Dai Fangao Zeng Yan Bai Jie Chang Yichen Wei 49 40 0 04 Aug 2020
Bootstrap your own latent: A new approach to self-supervised Learning Jean-Bastien Grill Florian Strub Florent Altché Corentin Tallec Pierre Harvey Richemond ... M. G. Azar Bilal Piot Koray Kavukcuoglu Rémi Munos Michal Valko SSL 371 6,806 0 13 Jun 2020
Knowledge Distillation Meets Self-Supervision Guodong Xu Ziwei Liu Xiaoxiao Li Chen Change Loy FedML 79 285 0 12 Jun 2020
Contrastive Representation Distillation Yonglong Tian Dilip Krishnan Phillip Isola 151 1,049 0 23 Oct 2019
On the Efficacy of Knowledge Distillation Ligang He Rui Mao 94 609 0 03 Oct 2019
Similarity-Preserving Knowledge Distillation Frederick Tung Greg Mori 124 978 0 23 Jul 2019
Contrastive Multiview Coding Yonglong Tian Dilip Krishnan Phillip Isola SSL 171 2,409 0 13 Jun 2019
Similarity of Neural Network Representations Revisited Simon Kornblith Mohammad Norouzi Honglak Lee Geoffrey E. Hinton 141 1,429 0 01 May 2019
Relational Knowledge Distillation Wonpyo Park Dongju Kim Yan Lu Minsu Cho 68 1,420 0 10 Apr 2019
A Comprehensive Overhaul of Feature Distillation Byeongho Heo Jeesoo Kim Sangdoo Yun Hyojin Park Nojun Kwak J. Choi 83 582 0 03 Apr 2019
Correlation Congruence for Knowledge Distillation Baoyun Peng Xiao Jin Jiaheng Liu Shunfeng Zhou Yichao Wu Yu Liu Dongsheng Li Zhaoning Zhang 89 512 0 03 Apr 2019
MobileNetV2: Inverted Residuals and Linear Bottlenecks Mark Sandler Andrew G. Howard Menglong Zhu A. Zhmoginov Liang-Chieh Chen 184 19,316 0 13 Jan 2018
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices Xiangyu Zhang Xinyu Zhou Mengxiao Lin Jian Sun AI4TS 141 6,878 0 04 Jul 2017
Deep Mutual Learning Ying Zhang Tao Xiang Timothy M. Hospedales Huchuan Lu FedML 151 1,653 0 01 Jun 2017
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications Andrew G. Howard Menglong Zhu Bo Chen Dmitry Kalenichenko Weijun Wang Tobias Weyand M. Andreetto Hartwig Adam 3DH 1.2K 20,858 0 17 Apr 2017
Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer Sergey Zagoruyko N. Komodakis 127 2,583 0 12 Dec 2016
Aggregated Residual Transformations for Deep Neural Networks Saining Xie Ross B. Girshick Piotr Dollár Zhuowen Tu Kaiming He 522 10,330 0 16 Nov 2016
Densely Connected Convolutional Networks Gao Huang Zhuang Liu Laurens van der Maaten Kilian Q. Weinberger PINN 3DV 775 36,861 0 25 Aug 2016
Gaussian Error Linear Units (GELUs) Dan Hendrycks Kevin Gimpel 172 5,011 0 27 Jun 2016
Wide Residual Networks Sergey Zagoruyko N. Komodakis 349 7,985 0 23 May 2016
End to End Learning for Self-Driving Cars Mariusz Bojarski D. Testa Daniel Dworakowski Bernhard Firner B. Flepp ... Urs Muller Jiakai Zhang Xin Zhang Jake Zhao Karol Zieba SSL 100 4,170 0 25 Apr 2016
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification Kaiming He Xinming Zhang Shaoqing Ren Jian Sun VLM 326 18,625 0 06 Feb 2015
FitNets: Hints for Thin Deep Nets Adriana Romero Nicolas Ballas Samira Ebrahimi Kahou Antoine Chassang C. Gatta Yoshua Bengio FedML 308 3,893 0 19 Dec 2014
Going Deeper with Convolutions Christian Szegedy Wei Liu Yangqing Jia P. Sermanet Scott E. Reed Dragomir Anguelov D. Erhan Vincent Vanhoucke Andrew Rabinovich 477 43,685 0 17 Sep 2014
ImageNet Large Scale Visual Recognition Challenge Olga Russakovsky Jia Deng Hao Su J. Krause S. Satheesh ... A. Karpathy A. Khosla Michael S. Bernstein Alexander C. Berg Li Fei-Fei VLM ObjD 1.7K 39,547 0 01 Sep 2014
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks Andrew M. Saxe James L. McClelland Surya Ganguli ODL 178 1,849 0 20 Dec 2013