DearKD: Data-Efficient Early Knowledge Distillation for Vision Transformers

27 April 2022

Xianing Chen

Qiong Cao

Yujie Zhong

Jing Zhang

Papers citing "DearKD: Data-Efficient Early Knowledge Distillation for Vision Transformers"

43 / 43 papers shown

Title
MoKD: Multi-Task Optimization for Knowledge Distillation Zeeshan Hayder A. Cheraghian Lars Petersson Mehrtash Harandi VLM 54 0 0 13 May 2025
Image Recognition with Online Lightweight Vision Transformer: A Survey Zherui Zhang Rongtao Xu Jie Zhou Changwei Wang Xingtian Pei ... Jiguang Zhang Li Guo Longxiang Gao Wenyuan Xu Shibiao Xu ViT 148 0 0 06 May 2025
Delving Deep into Semantic Relation Distillation Zhaoyi Yan Kangjun Liu Qixiang Ye 54 0 0 27 Mar 2025
CustomKD: Customizing Large Vision Foundation for Edge Model Improvement via Knowledge Distillation Jungsoo Lee Debasmit Das Munawar Hayat Sungha Choi Kyuwoong Hwang Fatih Porikli VLM 68 1 0 23 Mar 2025
Mix-QViT: Mixed-Precision Vision Transformer Quantization Driven by Layer Importance and Quantization Sensitivity Navin Ranjan Andreas E. Savakis MQ 47 1 0 10 Jan 2025
Computer Vision Model Compression Techniques for Embedded Systems: A Survey Alexandre Lopes Fernando Pereira dos Santos D. Oliveira Mauricio Schiezaro Hélio Pedrini 31 5 0 15 Aug 2024
An Attention-based Representation Distillation Baseline for Multi-Label Continual Learning Martin Menabue Emanuele Frascaroli Matteo Boschini Lorenzo Bonicelli Angelo Porrello Simone Calderara CLL 23 0 0 19 Jul 2024
Neural-based Video Compression on Solar Dynamics Observatory Images Atefeh Khoshkhahtinat Ali Zafari P. Mehta Nasser M. Nasrabadi Barbara J. Thompson M. Kirk D. D. Silva 48 0 0 12 Jul 2024
HDKD: Hybrid Data-Efficient Knowledge Distillation Network for Medical Image Classification Omar S. El-Assiouti Ghada Hamed Dina Khattab H. M. Ebied 42 1 0 10 Jul 2024
CTRL-F: Pairing Convolution with Transformer for Image Classification via Multi-Level Feature Cross-Attention and Representation Learning Fusion Hosam S. El-Assiouti Hadeer El-Saadawy M. Al-Berry M. Tolba ViT 52 0 0 09 Jul 2024
AMD: Automatic Multi-step Distillation of Large-scale Vision Models Cheng Han Qifan Wang S. Dianat Majid Rabbani Raghuveer M. Rao Yi Fang Qiang Guan Lifu Huang Dongfang Liu VLM 35 4 0 05 Jul 2024
Adaptive Modality Balanced Online Knowledge Distillation for Brain-Eye-Computer based Dim Object Detection Zixing Li Chao Yan Zhen Lan Xiaojia Xiang Han Zhou Jun Lai Dengqing Tang 43 0 0 02 Jul 2024
Efficient Multimodal Large Language Models: A Survey Yizhang Jin Jian Li Yexin Liu Tianjun Gu Kai Wu ... Xin Tan Zhenye Gan Yabiao Wang Chengjie Wang Lizhuang Ma LRM 47 45 0 17 May 2024
Data-independent Module-aware Pruning for Hierarchical Vision Transformers Yang He Qiufeng Wang ViT 50 3 0 21 Apr 2024
Learning to Project for Cross-Task Knowledge Distillation Dylan Auty Roy Miles Benedikt Kolbeinsson K. Mikolajczyk 40 0 0 21 Mar 2024
$V_kD:$ Improving Knowledge Distillation using Orthogonal Projections Roy Miles Ismail Elezi Jiankang Deng 52 10 0 10 Mar 2024
A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition Yusheng Dai Hang Chen Jun Du Ruoyu Wang Shihao Chen Jie Ma Haotian Wang Chin-Hui Lee 43 4 0 07 Mar 2024
Logit Standardization in Knowledge Distillation Shangquan Sun Wenqi Ren Jingzhi Li Rui Wang Xiaochun Cao 37 56 0 03 Mar 2024
A Survey on Transformer Compression Yehui Tang Yunhe Wang Jianyuan Guo Zhijun Tu Kai Han Hailin Hu Dacheng Tao 34 28 0 05 Feb 2024
LRP-QViT: Mixed-Precision Vision Transformer Quantization via Layer-wise Relevance Propagation Navin Ranjan Andreas E. Savakis MQ 26 6 0 20 Jan 2024
Cloud-Device Collaborative Learning for Multimodal Large Language Models Guanqun Wang Jiaming Liu Chenxuan Li Junpeng Ma Yuan Zhang ... Kevin Zhang Maurice Chong Ray Zhang Yijiang Liu Shanghang Zhang 41 7 0 26 Dec 2023
RdimKD: Generic Distillation Paradigm by Dimensionality Reduction Yi Guo Yiqian He Xiaoyang Li Haotong Qin Van Tung Pham Yang Zhang Shouda Liu 43 1 0 14 Dec 2023
Improved TokenPose with Sparsity Anning Li ViT 34 0 0 16 Nov 2023
Quantized Distillation: Optimizing Driver Activity Recognition Models for Resource-Constrained Environments Calvin Tanama Kunyu Peng Zdravko Marinov Rainer Stiefelhagen Alina Roitberg 17 1 0 10 Nov 2023
YFlows: Systematic Dataflow Exploration and Code Generation for Efficient Neural Network Inference using SIMD Architectures on CPUs Cyrus Zhou Zack Hassman Ruize Xu Dhirpal Shah Vaughn Richard Yanjing Li 32 1 0 01 Oct 2023
A survey on efficient vision transformers: algorithms, techniques, and performance benchmarking Lorenzo Papa Paolo Russo Irene Amerini Luping Zhou 30 42 0 05 Sep 2023
Reverse Knowledge Distillation: Training a Large Model using a Small One for Retinal Image Matching on Limited Data Sahar Almahfouz Nasser N. Gupte A. Sethi MedIm 17 12 0 20 Jul 2023
Cumulative Spatial Knowledge Distillation for Vision Transformers Borui Zhao Renjie Song Jiajun Liang 26 14 0 17 Jul 2023
A Survey of Techniques for Optimizing Transformer Inference Krishna Teja Chitty-Venkata Sparsh Mittal M. Emani V. Vishwanath Arun Somani 43 62 0 16 Jul 2023
Multimodal Distillation for Egocentric Action Recognition Gorjan Radevski Dusan Grujicic Marie-Francine Moens Matthew Blaschko Tinne Tuytelaars EgoV 23 23 0 14 Jul 2023
Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation Chuanguang Yang Xinqiang Yu Zhulin An Yongjun Xu VLM OffRL 86 22 0 19 Jun 2023
VanillaKD: Revisit the Power of Vanilla Knowledge Distillation from Small Scale to Large Scale Zhiwei Hao Jianyuan Guo Kai Han Han Hu Chang Xu Yunhe Wang 35 16 0 25 May 2023
HARD: Hard Augmentations for Robust Distillation Arne F. Nix Max F. Burg Fabian H. Sinz AAML 33 1 0 24 May 2023
Distilling Token-Pruned Pose Transformer for 2D Human Pose Estimation Feixiang Ren ViT 19 2 0 12 Apr 2023
Understanding the Role of the Projector in Knowledge Distillation Roy Miles K. Mikolajczyk 24 21 0 20 Mar 2023
Structured Pruning for Deep Convolutional Neural Networks: A survey Yang He Lingao Xiao 3DPC 30 117 0 01 Mar 2023
Generic-to-Specific Distillation of Masked Autoencoders Wei Huang Zhiliang Peng Li Dong Furu Wei Jianbin Jiao QiXiang Ye 32 22 0 28 Feb 2023
Knowledge Distillation in Vision Transformers: A Critical Review Gousia Habib Tausifa Jan Saleem Brejesh Lall 29 15 0 04 Feb 2023
PPT: token-Pruned Pose Transformer for monocular and multi-view human pose estimation Haoyu Ma Zhe Wang Yifei Chen Deying Kong Liangjian Chen Xingwei Liu Xiangyi Yan Hao Tang Xiaohui Xie ViT 35 47 0 16 Sep 2022
Masked Autoencoders Enable Efficient Knowledge Distillers Yutong Bai Zeyu Wang Junfei Xiao Chen Wei Huiyu Wang Alan Yuille Yuyin Zhou Cihang Xie CLL 29 39 0 25 Aug 2022
ReAct: Temporal Action Detection with Relational Queries Ding Shi Yujie Zhong Qiong Cao Jing Zhang Lin Ma Jia Li Dacheng Tao ViT 28 68 0 14 Jul 2022
The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillation Zihui Xue Zhengqi Gao Sucheng Ren Hang Zhao 24 37 0 13 Jun 2022
OH-Former: Omni-Relational High-Order Transformer for Person Re-Identification Xianing Chen Chunlin Xu Qiong Cao Jialang Xu Yujie Zhong Jiale Xu Zhengxin Li Jingya Wang Shenghua Gao ViT 71 18 0 23 Sep 2021