v1v2v3 (latest)

Knowledge Distillation from A Stronger Teacher

21 May 2022

Fei Wang

ArXiv (abs)PDF HTML Github (146★)

Papers citing "Knowledge Distillation from A Stronger Teacher"

31 / 131 papers shown

Title
Talking Models: Distill Pre-trained Knowledge to Downstream Models via Interactive Communication Zhe Zhao Qingyun Liu Huan Gui Bang An Lichan Hong Ed H. Chi 77 1 0 04 Oct 2023
Improving Knowledge Distillation with Teacher's Explanation S. Chowdhury Ben Liang A. Tizghadam Ilijc Albanese FAtt 30 0 0 04 Oct 2023
GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graph Xin Li Dongze Lian Zhihe Lu Jiawang Bai Zhibo Chen Xinchao Wang VLM 113 66 0 24 Sep 2023
Heterogeneous Generative Knowledge Distillation with Masked Image Modeling Ziming Wang Shumin Han Xiaodi Wang Jing Hao Xianbin Cao Baochang Zhang VLM 74 0 0 18 Sep 2023
Teacher-Student Architecture for Knowledge Distillation: A Survey Chengming Hu Xuan Li Danyang Liu Haolun Wu Xi Chen Ju Wang Xue Liu 94 19 0 08 Aug 2023
TeachCLIP: Multi-Grained Teaching for Efficient Text-to-Video Retrieval Kaibin Tian Rui Zhao Hu Hu Runquan Xie Fengzong Lian Zhanhui Kang Xirong Li CLIP 47 0 0 02 Aug 2023
Effective Whole-body Pose Estimation with Two-stages Distillation Zhendong Yang Ailing Zeng Chun Yuan Yu Li 141 183 0 29 Jul 2023
Audio Embeddings as Teachers for Music Classification Yiwei Ding Alexander Lerch 65 5 0 30 Jun 2023
Positive Label Is All You Need for Multi-Label Classification Zhixiang Yuan Kai Zhang Tao Huang NoLa 102 6 0 28 Jun 2023
Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation Chuanguang Yang Xinqiang Yu Zhulin An Yongjun Xu VLM OffRL 198 27 0 19 Jun 2023
Exploring the Application of Large-scale Pre-trained Models on Adverse Weather Removal Zhentao Tan Yue-bo Wu Qiankun Liu Qi Chu Le Lu Jieping Ye Nenghai Yu 95 13 0 15 Jun 2023
Are Large Kernels Better Teachers than Transformers for ConvNets? Tianjin Huang Lu Yin Zhenyu Zhang Lijuan Shen Meng Fang Mykola Pechenizkiy Zhangyang Wang Shiwei Liu 95 13 0 30 May 2023
Improving Knowledge Distillation via Regularizing Feature Norm and Direction Yuzhu Wang Lechao Cheng Manni Duan Yongheng Wang Zunlei Feng Shu Kong 95 22 0 26 May 2023
VanillaKD: Revisit the Power of Vanilla Knowledge Distillation from Small Scale to Large Scale Zhiwei Hao Jianyuan Guo Kai Han Han Hu Chang Xu Yunhe Wang 74 16 0 25 May 2023
Knowledge Diffusion for Distillation Tao Huang Yuan Zhang Mingkai Zheng Shan You Fei Wang Chao Qian Chang Xu 108 56 0 25 May 2023
How to Distill your BERT: An Empirical Study on the Impact of Weight Initialisation and Distillation Objectives Xinpeng Wang Leonie Weissweiler Hinrich Schütze Barbara Plank 68 8 0 24 May 2023
Decoupled Kullback-Leibler Divergence Loss Jiequan Cui Zhuotao Tian Zhisheng Zhong Xiaojuan Qi Bei Yu Hanwang Zhang 83 46 0 23 May 2023
Student-friendly Knowledge Distillation Mengyang Yuan Bo Lang Fengnan Quan 92 21 0 18 May 2023
Tailoring Instructions to Student's Learning Levels Boosts Knowledge Distillation Yuxin Ren Zi-Qi Zhong Xingjian Shi Yi Zhu Chun Yuan Mu Li 105 7 0 16 May 2023
Towards Effective Collaborative Learning in Long-Tailed Recognition Zhengzhuo Xu Zenghao Chai Chengying Xu Chun Yuan Haiqin Yang 82 6 0 05 May 2023
Avatar Knowledge Distillation: Self-ensemble Teacher Paradigm with Uncertainty Yuan Zhang Weihua Chen Yichen Lu Tao Huang Xiuyu Sun Jian Cao 120 8 0 04 May 2023
Towards Efficient Task-Driven Model Reprogramming with Foundation Models Shoukai Xu Jiangchao Yao Ran Luo Shuhai Zhang Zihao Lian Mingkui Tan Bo Han Yaowei Wang 95 6 0 05 Apr 2023
Dice Semimetric Losses: Optimizing the Dice Score with Soft Labels Zifu Wang Teodora Popordanoska J. Bertels Robin Lemmens Matthew B. Blaschko 60 10 0 28 Mar 2023
From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labels Zhendong Yang Ailing Zeng Zhe Li Tianke Zhang Chun Yuan Yu Li 115 80 0 23 Mar 2023
Knowledge Distillation from Single to Multi Labels: an Empirical Study Youcai Zhang Yuzhuo Qin Heng-Ye Liu Yanhao Zhang Yaqian Li X. Gu VLM 82 2 0 15 Mar 2023
Jaccard Metric Losses: Optimizing the Jaccard Index with Soft Labels Zifu Wang Xuefei Ning Matthew B. Blaschko VLM 114 15 0 11 Feb 2023
Knowledge Distillation in Vision Transformers: A Critical Review Gousia Habib Tausifa Jan Saleem Brejesh Lall 98 16 0 04 Feb 2023
A Simple Recipe for Competitive Low-compute Self supervised Vision Models Quentin Duval Ishan Misra Nicolas Ballas 70 9 0 23 Jan 2023
TransKD: Transformer Knowledge Distillation for Efficient Semantic Segmentation R. Liu Kailun Yang Alina Roitberg Jiaming Zhang Kunyu Peng Huayao Liu Yaonan Wang Rainer Stiefelhagen ViT 91 38 0 27 Feb 2022
Knowledge Distillation with Deep Supervision Shiya Luo Defang Chen Can Wang 65 2 0 16 Feb 2022
Contrastive Representation Distillation Yonglong Tian Dilip Krishnan Phillip Isola 270 1,059 0 23 Oct 2019