Attention Distillation: self-supervised vision transformer students need
more guidance

Attention Distillation: self-supervised vision transformer students need more guidance

3 October 2022

Joost van de Weijer

Papers citing "Attention Distillation: self-supervised vision transformer students need more guidance"

18 / 18 papers shown

Title
$Simple Semi-supervised Knowledge Distillation from Vision-Language Models via $\mathbf{\texttt{D}}$ual-$\mathbf{\texttt{H}}$ead $\mathbf{\texttt{O}}$ptimization$ Simple Semi-supervised Knowledge Distillation from Vision-Language Models via $\mathbf{\texttt{D}}$ ual- $\mathbf{\texttt{H}}$ ead $\mathbf{\texttt{O}}$ ptimization Seongjae Kang Dong Bok Lee Hyungjoon Jang Sung Ju Hwang VLM 57 0 0 12 May 2025
Image Recognition with Online Lightweight Vision Transformer: A Survey Zherui Zhang Rongtao Xu Jie Zhou Changwei Wang Xingtian Pei ... Jiguang Zhang Li Guo Longxiang Gao Wenyuan Xu Shibiao Xu ViT 148 0 0 06 May 2025
X2I: Seamless Integration of Multimodal Understanding into Diffusion Transformer via Attention Distillation Jian Ma Qirong Peng Xu Guo Chen Chen H. Lu Zhenyu Yang VLM 69 1 0 08 Mar 2025
Knowledge Distillation in RNN-Attention Models for Early Prediction of Student Performance Sukrit Leelaluk Cheng Tang Valdemar Švábenský Atsushi Shimada 67 1 0 19 Dec 2024
On the Surprising Effectiveness of Attention Transfer for Vision Transformers Alexander C. Li Yuandong Tian Bin Chen Deepak Pathak Xinlei Chen 43 0 0 14 Nov 2024
Simple Unsupervised Knowledge Distillation With Space Similarity Aditya Singh Haohan Wang 31 1 0 20 Sep 2024
A Survey of the Self Supervised Learning Mechanisms for Vision Transformers Asifullah Khan A. Sohail M. Fiaz Mehdi Hassan Tariq Habib Afridi ... Muhammad Zaigham Zaheer Kamran Ali Tangina Sultana Ziaurrehman Tanoli Naeem Akhter 45 3 0 30 Aug 2024
Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation Chuanguang Yang Xinqiang Yu Zhulin An Yongjun Xu VLM OffRL 86 22 0 19 Jun 2023
KS-DETR: Knowledge Sharing in Attention Learning for Detection Transformer Kaikai Zhao Norimichi Ukita MU 43 1 0 22 Feb 2023
Masked Autoencoders Are Scalable Vision Learners Kaiming He Xinlei Chen Saining Xie Yanghao Li Piotr Dollár Ross B. Girshick ViT TPM 305 7,443 0 11 Nov 2021
MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer Sachin Mehta Mohammad Rastegari ViT 218 1,213 0 05 Oct 2021
Mobile-Former: Bridging MobileNet and Transformer Yinpeng Chen Xiyang Dai Dongdong Chen Mengchen Liu Xiaoyi Dong Lu Yuan Zicheng Liu ViT 180 476 0 12 Aug 2021
Emerging Properties in Self-Supervised Vision Transformers Mathilde Caron Hugo Touvron Ishan Misra Hervé Jégou Julien Mairal Piotr Bojanowski Armand Joulin 320 5,785 0 29 Apr 2021
Distilling Knowledge via Knowledge Review Pengguang Chen Shu-Lin Liu Hengshuang Zhao Jiaya Jia 152 422 0 19 Apr 2021
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions Wenhai Wang Enze Xie Xiang Li Deng-Ping Fan Kaitao Song Ding Liang Tong Lu Ping Luo Ling Shao ViT 283 3,623 0 24 Feb 2021
SEED: Self-supervised Distillation For Visual Representation Zhiyuan Fang Jianfeng Wang Lijuan Wang Lei Zhang Yezhou Yang Zicheng Liu SSL 242 190 0 12 Jan 2021
Boosting Self-Supervised Learning via Knowledge Transfer M. Noroozi Ananth Vinjimoor Paolo Favaro Hamed Pirsiavash SSL 209 292 0 01 May 2018
ImageNet Large Scale Visual Recognition Challenge Olga Russakovsky Jia Deng Hao Su J. Krause S. Satheesh ... A. Karpathy A. Khosla Michael S. Bernstein Alexander C. Berg Li Fei-Fei VLM ObjD 296 39,198 0 01 Sep 2014