TinyMIM: An Empirical Study of Distilling MIM Pre-trained Models

3 January 2023

ArXiv (abs)PDF HTML Github (156★)

Papers citing "TinyMIM: An Empirical Study of Distilling MIM Pre-trained Models"

50 / 61 papers shown

Title
UNIP: Rethinking Pre-trained Attention Patterns for Infrared Semantic Segmentation Tao Zhang Jinyong Wen Zhen Chen Kun Ding Di Zhang Chunhong Pan 233 1 0 04 Feb 2025
How Well Do Supervised 3D Models Transfer to Medical Imaging Tasks? Wenxuan Li Alan Yuille Zongwei Zhou MedIm 128 10 0 20 Jan 2025
Dataset Distillation via Factorization Songhua Liu Kai Wang Xingyi Yang Jingwen Ye Xinchao Wang DD 191 144 0 30 Oct 2022
MobileViTv3: Mobile-Friendly Vision Transformer with Simple and Effective Fusion of Local, Global and Input Features S. Wadekar Abhishek Chaurasia ViT 125 92 0 30 Sep 2022
BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers Zhiliang Peng Li Dong Hangbo Bao QiXiang Ye Furu Wei 66 319 0 12 Aug 2022
MILAN: Masked Image Pretraining on Language Assisted Representation Zejiang Hou Fei Sun Yen-kuang Chen Yuan Xie S. Kung ViT 75 68 0 11 Aug 2022
SdAE: Self-distillated Masked Autoencoder Yabo Chen Yuchen Liu Dongsheng Jiang Xiaopeng Zhang Wenrui Dai H. Xiong Qi Tian ViT 70 73 0 31 Jul 2022
Factorizing Knowledge in Neural Networks Xingyi Yang Jingwen Ye Xinchao Wang MoMe 85 124 0 04 Jul 2022
The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillation Zihui Xue Zhengqi Gao Sucheng Ren Hang Zhao 86 40 0 13 Jun 2022
On Data Scaling in Masked Image Modeling Zhenda Xie Zheng Zhang Yue Cao Yutong Lin Yixuan Wei Qi Dai Han Hu 76 55 0 09 Jun 2022
Siamese Image Modeling for Self-Supervised Vision Representation Learning Chenxin Tao Xizhou Zhu Weijie Su Gao Huang Bin Li Jie Zhou Yu Qiao Xiaogang Wang Jifeng Dai SSL 88 96 0 02 Jun 2022
EfficientFormer: Vision Transformers at MobileNet Speed Yanyu Li Geng Yuan Yang Wen Eric Hu Georgios Evangelidis Sergey Tulyakov Yanzhi Wang Jian Ren ViT 87 367 0 02 Jun 2022
Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation Yixuan Wei Han Hu Zhenda Xie Zheng Zhang Yue Cao Jianmin Bao Dong Chen B. Guo CLIP 134 126 0 27 May 2022
EdgeViTs: Competing Light-weight CNNs on Mobile Devices with Vision Transformers Junting Pan Adrian Bulat Fuwen Tan Xiatian Zhu Łukasz Dudziak Hongsheng Li Georgios Tzimiropoulos Brais Martínez ViT 82 190 0 06 May 2022
data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language Alexei Baevski Wei-Ning Hsu Qiantong Xu Arun Babu Jiatao Gu Michael Auli SSL VLM ViT 97 858 0 07 Feb 2022
Corrupted Image Modeling for Self-Supervised Visual Pre-Training Yuxin Fang Li Dong Hangbo Bao Xinggang Wang Furu Wei 66 88 0 07 Feb 2022
Context Autoencoder for Self-Supervised Representation Learning Xiaokang Chen Mingyu Ding Xiaodi Wang Ying Xin Shentong Mo Yunhao Wang Shumin Han Ping Luo Gang Zeng Jingdong Wang SSL 105 396 0 07 Feb 2022
Masked Feature Prediction for Self-Supervised Visual Pre-Training Chen Wei Haoqi Fan Saining Xie Chaoxia Wu Alan Yuille Christoph Feichtenhofer ViT 145 668 0 16 Dec 2021
Safe Distillation Box Jingwen Ye Yining Mao Mingli Song Xinchao Wang Cheng Jin Xiuming Zhang AAML 95 14 0 05 Dec 2021
Shunted Self-Attention via Multi-Scale Token Aggregation Sucheng Ren Daquan Zhou Shengfeng He Jiashi Feng Xinchao Wang ViT 82 227 0 30 Nov 2021
PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers Xiaoyi Dong Jianmin Bao Ting Zhang Dongdong Chen Weiming Zhang Lu Yuan Dong Chen Fang Wen Nenghai Yu Baining Guo ViT 105 243 0 24 Nov 2021
SimMIM: A Simple Framework for Masked Image Modeling Zhenda Xie Zheng Zhang Yue Cao Yutong Lin Jianmin Bao Zhuliang Yao Qi Dai Han Hu 197 1,353 0 18 Nov 2021
Swin Transformer V2: Scaling Up Capacity and Resolution Ze Liu Han Hu Yutong Lin Zhuliang Yao Zhenda Xie ... Yue Cao Zheng Zhang Li Dong Furu Wei B. Guo ViT 215 1,813 0 18 Nov 2021
iBOT: Image BERT Pre-Training with Online Tokenizer Jinghao Zhou Chen Wei Huiyu Wang Wei Shen Cihang Xie Alan Yuille Tao Kong 81 735 0 15 Nov 2021
Masked Autoencoders Are Scalable Vision Learners Kaiming He Xinlei Chen Saining Xie Yanghao Li Piotr Dollár Ross B. Girshick ViT TPM 465 7,757 0 11 Nov 2021
MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer Sachin Mehta Mohammad Rastegari ViT 285 1,274 0 05 Oct 2021
An Information Theory-inspired Strategy for Automatic Network Pruning Xiawu Zheng Yuexiao Ma Teng Xi Gang Zhang Errui Ding Yuchao Li Jie Chen Yonghong Tian Rongrong Ji 173 13 0 19 Aug 2021
Mobile-Former: Bridging MobileNet and Transformer Yinpeng Chen Xiyang Dai Dongdong Chen Mengchen Liu Xiaoyi Dong Lu Yuan Zicheng Liu ViT 258 488 0 12 Aug 2021
Co-advise: Cross Inductive Bias Distillation Sucheng Ren Zhengqi Gao Tianyu Hua Zihui Xue Yonglong Tian Shengfeng He Hang Zhao 74 52 0 23 Jun 2021
BEiT: BERT Pre-Training of Image Transformers Hangbo Bao Li Dong Songhao Piao Furu Wei ViT 274 2,826 0 15 Jun 2021
Emerging Properties in Self-Supervised Vision Transformers Mathilde Caron Hugo Touvron Ishan Misra Hervé Jégou Julien Mairal Piotr Bojanowski Armand Joulin 694 6,079 0 29 Apr 2021
An Empirical Study of Training Self-Supervised Vision Transformers Xinlei Chen Saining Xie Kaiming He ViT 154 1,864 0 05 Apr 2021
LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference Ben Graham Alaaeldin El-Nouby Hugo Touvron Pierre Stock Armand Joulin Hervé Jégou Matthijs Douze ViT 78 788 0 02 Apr 2021
Multimodal Knowledge Expansion Zihui Xue Sucheng Ren Zhengqi Gao Hang Zhao 113 30 0 26 Mar 2021
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows Ze Liu Yutong Lin Yue Cao Han Hu Yixuan Wei Zheng Zhang Stephen Lin B. Guo ViT 453 21,439 0 25 Mar 2021
Learning Transferable Visual Models From Natural Language Supervision Alec Radford Jong Wook Kim Chris Hallacy Aditya A. Ramesh Gabriel Goh ... Amanda Askell Pamela Mishkin Jack Clark Gretchen Krueger Ilya Sutskever CLIP VLM 929 29,436 0 26 Feb 2021
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions Wenhai Wang Enze Xie Xiang Li Deng-Ping Fan Kaitao Song Ding Liang Tong Lu Ping Luo Ling Shao ViT 530 3,724 0 24 Feb 2021
Zero-Shot Text-to-Image Generation Aditya A. Ramesh Mikhail Pavlov Gabriel Goh Scott Gray Chelsea Voss Alec Radford Mark Chen Ilya Sutskever VLM 415 4,953 0 24 Feb 2021
SEED: Self-supervised Distillation For Visual Representation Zhiyuan Fang Jianfeng Wang Lijuan Wang Lei Zhang Yezhou Yang Zicheng Liu SSL 298 194 0 12 Jan 2021
Training data-efficient image transformers & distillation through attention Hugo Touvron Matthieu Cord Matthijs Douze Francisco Massa Alexandre Sablayrolles Hervé Jégou ViT 387 6,768 0 23 Dec 2020
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai ... Matthias Minderer G. Heigold Sylvain Gelly Jakob Uszkoreit N. Houlsby ViT 657 41,103 0 22 Oct 2020
The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization Dan Hendrycks Steven Basart Norman Mu Saurav Kadavath Frank Wang ... Samyak Parajuli Mike Guo Basel Alomair Jacob Steinhardt Justin Gilmer OOD 340 1,740 0 29 Jun 2020
Knowledge Distillation: A Survey Jianping Gou B. Yu Stephen J. Maybank Dacheng Tao VLM 105 2,960 0 09 Jun 2020
Data-Free Knowledge Amalgamation via Group-Stack Dual-GAN Jingwen Ye Yixin Ji Xinchao Wang Xin Gao Xiuming Zhang 93 53 0 20 Mar 2020
On the Efficacy of Knowledge Distillation Ligang He Rui Mao 94 609 0 03 Oct 2019
Natural Adversarial Examples Dan Hendrycks Kevin Zhao Steven Basart Jacob Steinhardt Basel Alomair OODD 212 1,472 0 16 Jul 2019
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks Mingxing Tan Quoc V. Le 3DV MedIm 142 18,134 0 28 May 2019
Searching for MobileNetV3 Andrew G. Howard Mark Sandler Grace Chu Liang-Chieh Chen Bo Chen ... Yukun Zhu Ruoming Pang Vijay Vasudevan Quoc V. Le Hartwig Adam 359 6,782 0 06 May 2019
A Comprehensive Overhaul of Feature Distillation Byeongho Heo Jeesoo Kim Sangdoo Yun Hyojin Park Nojun Kwak J. Choi 83 580 0 03 Apr 2019
Benchmarking Neural Network Robustness to Common Corruptions and Perturbations Dan Hendrycks Thomas G. Dietterich OOD VLM 174 3,435 0 28 Mar 2019