Knowledge Distillation as Efficient Pre-training: Faster Convergence,
Higher Data-efficiency, and Better Transferability

Knowledge Distillation as Efficient Pre-training: Faster Convergence, Higher Data-efficiency, and Better Transferability

10 March 2022

Shuyang Sun

Xiaojuan Qi

Papers citing "Knowledge Distillation as Efficient Pre-training: Faster Convergence, Higher Data-efficiency, and Better Transferability"

18 / 18 papers shown

Title
Reinforced Model Merging J. N. Han Jingwen Ye Shunyu Liu Haofei Zhang Jie Song Zunlei Feng Mingli Song MoMe 55 0 0 27 Mar 2025
Keeping Representation Similarity in Finetuning for Medical Image Analysis Wenqiang Zu Shenghao Xie Hao Chen Yiming Liang Lei Ma MedIm OOD 48 0 0 10 Mar 2025
Frequency-Guided Masking for Enhanced Vision Self-Supervised Learning Amin Karimi Monsefi Mengxi Zhou Nastaran Karimi Monsefi Ser-Nam Lim Wei-Lun Chao R. Ramnath 46 1 0 16 Sep 2024
Lightweight Model Pre-training via Language Guided Knowledge Distillation Mingsheng Li Lin Zhang Mingzhen Zhu Zilong Huang Gang Yu Jiayuan Fan Tao Chen 41 1 0 17 Jun 2024
Acceleration Algorithms in GNNs: A Survey Lu Ma Zeang Sheng Xunkai Li Xin Gao Zhezheng Hao Ling Yang Wentao Zhang Bin Cui GNN 42 3 0 07 May 2024
Zero-Shot Distillation for Image Encoders: How to Make Effective Use of Synthetic Data Niclas Popp J. H. Metzen Matthias Hein VLM 42 1 0 25 Apr 2024
SeD: Semantic-Aware Discriminator for Image Super-Resolution Bingchen Li Xin Li Hanxin Zhu Yeying Jin Ruoyu Feng Zhizheng Zhang Zhibo Chen SupR 48 22 0 29 Feb 2024
A Deep Hierarchical Feature Sparse Framework for Occluded Person Re-Identification Yihu Song Shuaishi Liu 33 1 0 15 Jan 2024
Quantized Distillation: Optimizing Driver Activity Recognition Models for Resource-Constrained Environments Calvin Tanama Kunyu Peng Zdravko Marinov Rainer Stiefelhagen Alina Roitberg 19 1 0 10 Nov 2023
Online Speculative Decoding Xiaoxuan Liu Lanxiang Hu Peter Bailis Alvin Cheung Zhijie Deng Ion Stoica Hao Zhang 27 51 0 11 Oct 2023
Towards a Smaller Student: Capacity Dynamic Distillation for Efficient Image Retrieval Yi Xie Huaidong Zhang Xuemiao Xu Jianqing Zhu Shengfeng He VLM 21 13 0 16 Mar 2023
TAKT: Target-Aware Knowledge Transfer for Whole Slide Image Classification Conghao Xiong Yi-Mou Lin Hao Chen Hao Zheng Dong Wei Yefeng Zheng Joseph J. Y. Sung Irwin King 31 3 0 10 Mar 2023
Knowledge Distillation on Graphs: A Survey Yijun Tian Shichao Pei Xiangliang Zhang Chuxu Zhang Nitesh V. Chawla 21 28 0 01 Feb 2023
Long-Range Zero-Shot Generative Deep Network Quantization Yan Luo Yangcheng Gao Zhao Zhang Haijun Zhang Mingliang Xu Meng Wang MQ 31 9 0 13 Nov 2022
Attention Distillation: self-supervised vision transformer students need more guidance Kai Wang Fei Yang Joost van de Weijer ViT 30 16 0 03 Oct 2022
ImageNet-21K Pretraining for the Masses T. Ridnik Emanuel Ben-Baruch Asaf Noy Lihi Zelnik-Manor SSeg VLM CLIP 181 689 0 22 Apr 2021
Meta Pseudo Labels Hieu H. Pham Zihang Dai Qizhe Xie Minh-Thang Luong Quoc V. Le VLM 262 656 0 23 Mar 2020
Semantic Understanding of Scenes through the ADE20K Dataset Bolei Zhou Hang Zhao Xavier Puig Tete Xiao Sanja Fidler Adela Barriuso Antonio Torralba SSeg 253 1,829 0 18 Aug 2016