Training data-efficient image transformers & distillation through attention

23 December 2020

Alexandre Sablayrolles

Hervé Jégou

ViT

ArXiv PDF HTML

Papers citing "Training data-efficient image transformers & distillation through attention"

50 / 1,257 papers shown

Title
BasicTAD: an Astounding RGB-Only Baseline for Temporal Action Detection Mingdong Yang Guo Chen Yin-Dong Zheng Tong Lu Limin Wang 39 45 0 05 May 2022
Spot-adaptive Knowledge Distillation Jie Song Ying Chen Jingwen Ye Mingli Song 20 72 0 05 May 2022
Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completion Xiang Chen Ningyu Zhang Lei Li Shumin Deng Chuanqi Tan Changliang Xu Fei Huang Luo Si Huajun Chen 18 126 0 04 May 2022
Sequencer: Deep LSTM for Image Classification Yuki Tatsunami Masato Taki VLM ViT 16 78 0 04 May 2022
Simpler is Better: off-the-shelf Continual Learning Through Pretrained Backbones Francesco Pelosin VLM 16 11 0 03 May 2022
Better plain ViT baselines for ImageNet-1k Lucas Beyer Xiaohua Zhai Alexander Kolesnikov ViT VLM 33 111 0 03 May 2022
A Comprehensive Survey of Image Augmentation Techniques for Deep Learning Mingle Xu Sook Yoon A. Fuentes D. Park VLM 27 397 0 03 May 2022
Source Domain Subset Sampling for Semi-Supervised Domain Adaptation in Semantic Segmentation Daehan Kim Min-seok Seo Jinsun Park Dong-Geol Choi TTA 40 3 0 30 Apr 2022
Coarse-to-Fine Video Denoising with Dual-Stage Spatial-Channel Transformer Wu Yun Mengshi Qi Chuanming Wang Huiyuan Fu Huadong Ma ViT 11 6 0 30 Apr 2022
Improving Transferability for Domain Adaptive Detection Transformers Kaixiong Gong Shuang Li Shugang Li Rui Zhang Chi Harold Liu Qiang Chen 57 34 0 29 Apr 2022
A Challenging Benchmark of Anime Style Recognition Haotang Li S. Guo Kailin Lyu Xiao Yang Tianchen Chen Jianqing Zhu Huanqiang Zeng CVBM 21 5 0 29 Apr 2022
Depth Estimation with Simplified Transformer John Yang Le An Anurag Dixit Jinkyu Koo Su Inn Park MDE 28 21 0 28 Apr 2022
DearKD: Data-Efficient Early Knowledge Distillation for Vision Transformers Xianing Chen Qiong Cao Yujie Zhong Jing Zhang Shenghua Gao Dacheng Tao ViT 37 76 0 27 Apr 2022
Masked Spectrogram Prediction For Self-Supervised Audio Pre-Training Dading Chong Helin Wang Peilin Zhou Qingcheng Zeng 39 65 0 27 Apr 2022
Improving the Transferability of Adversarial Examples with Restructure Embedded Patches Huipeng Zhou Yu-an Tan Yajie Wang Haoran Lyu Shan-Hung Wu Yuan-zhang Li ViT 19 4 0 27 Apr 2022
Boosting Adversarial Transferability of MLP-Mixer Haoran Lyu Yajie Wang Yu-an Tan Huipeng Zhou Yuhang Zhao Quan-xin Zhang AAML 27 1 0 26 Apr 2022
Deeper Insights into the Robustness of ViTs towards Common Corruptions Rui Tian Zuxuan Wu Qi Dai Han Hu Yu-Gang Jiang ViT AAML 21 4 0 26 Apr 2022
Fused Audio Instance and Representation for Respiratory Disease Detection Tuan Truong Matthias Lenga A. Serrurier Sadegh Mohammadi 16 0 0 22 Apr 2022
Residual Mixture of Experts Lemeng Wu Mengchen Liu Yinpeng Chen Dongdong Chen Xiyang Dai Lu Yuan MoE 22 36 0 20 Apr 2022
ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models Chunyuan Li Haotian Liu Liunian Harold Li Pengchuan Zhang J. Aneja ... Ping Jin Houdong Hu Zicheng Liu Yong Jae Lee Jianfeng Gao 29 145 0 19 Apr 2022
CTCNet: A CNN-Transformer Cooperation Network for Face Image Super-Resolution Guangwei Gao Zixiang Xu Juncheng Li Jian Yang T. Zeng Guo-Jun Qi CVBM ViT SupR 36 80 0 19 Apr 2022
VSA: Learning Varied-Size Window Attention in Vision Transformers Qiming Zhang Yufei Xu Jing Zhang Dacheng Tao 22 53 0 18 Apr 2022
Application of Transfer Learning and Ensemble Learning in Image-level Classification for Breast Histopathology Yuchao Zheng Chen Li Xiaomin Zhou Hao Chen Hao Xu ... Haiqing Zhang Xirong Li Hongzan Sun Xinyu Huang M. Grzegorzek 33 55 0 18 Apr 2022
Searching Intrinsic Dimensions of Vision Transformers Fanghui Xue Biao Yang Y. Qi Jack Xin ViT 36 2 0 16 Apr 2022
BYOL for Audio: Exploring Pre-trained General-purpose Audio Representations Daisuke Niizumi Daiki Takeuchi Yasunori Ohishi N. Harada K. Kashino SSL 36 53 0 15 Apr 2022
MiniViT: Compressing Vision Transformers with Weight Multiplexing Jinnian Zhang Houwen Peng Kan Wu Mengchen Liu Bin Xiao Jianlong Fu Lu Yuan ViT 28 123 0 14 Apr 2022
Neighborhood Attention Transformer Ali Hassani Steven Walton Jiacheng Li Shengjia Li Humphrey Shi ViT AI4TS 36 253 0 14 Apr 2022
Masked Siamese Networks for Label-Efficient Learning Mahmoud Assran Mathilde Caron Ishan Misra Piotr Bojanowski Florian Bordes Pascal Vincent Armand Joulin Michael G. Rabbat Nicolas Ballas SSL 28 310 0 14 Apr 2022
ViTOL: Vision Transformer for Weakly Supervised Object Localization Saurav Gupta Sourav Lakhotia Abhay Rawat Rahul Tallamraju WSOL 32 21 0 14 Apr 2022
TopFormer: Token Pyramid Transformer for Mobile Semantic Segmentation Wenqiang Zhang Zilong Huang Guozhong Luo Tao Chen Xinggang Wang Wenyu Liu Gang Yu Chunhua Shen ViT 22 199 0 12 Apr 2022
OutfitTransformer: Learning Outfit Representations for Fashion Recommendation Rohan Sarkar Navaneeth Bodla Mariya I. Vasileva Yen-Liang Lin Anu Beniwal Alan Lu Gérard Medioni 27 35 0 11 Apr 2022
Fashionformer: A simple, Effective and Unified Baseline for Human Fashion Segmentation and Recognition Shilin Xu Xiangtai Li Jingbo Wang Guangliang Cheng Yunhai Tong Dacheng Tao ViT 23 27 0 10 Apr 2022
Points to Patches: Enabling the Use of Self-Attention for 3D Shape Recognition Axel Berg Magnus Oskarsson Mark O'Connor 3DPC ViT 29 26 0 08 Apr 2022
DaViT: Dual Attention Vision Transformers Mingyu Ding Bin Xiao Noel Codella Ping Luo Jingdong Wang Lu Yuan ViT 48 240 0 07 Apr 2022
Solving ImageNet: a Unified Scheme for Training any Backbone to Top Results T. Ridnik Hussam Lawen Emanuel Ben-Baruch Asaf Noy 38 11 0 07 Apr 2022
An Empirical Study of Remote Sensing Pretraining Di Wang Jing Zhang Bo Du Guisong Xia Dacheng Tao EDL 36 190 0 06 Apr 2022
Towards An End-to-End Framework for Flow-Guided Video Inpainting Z. Li Cheng Lu Jia Qin Chunle Guo Mingg-Ming Cheng 41 149 0 06 Apr 2022
MixFormer: Mixing Features across Windows and Dimensions Qiang Chen Qiman Wu Jian Wang Qinghao Hu T. Hu Errui Ding Jian Cheng Jingdong Wang MDE ViT 31 101 0 06 Apr 2022
Vision Transformer Equipped with Neural Resizer on Facial Expression Recognition Task Hyeonbin Hwang Soyeon Kim Wei-Jin Park Jiho Seo Kyungtae Ko Hyeon Yeo ViT 45 9 0 05 Apr 2022
Region Rebalance for Long-Tailed Semantic Segmentation Jiequan Cui Yuhui Yuan Zhisheng Zhong Zhuotao Tian Han Hu Stephen Lin Jiaya Jia 18 18 0 05 Apr 2022
Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videos Shao-Wei Liu Subarna Tripathi Somdeb Majumdar Xiaolong Wang EgoV 29 93 0 04 Apr 2022
MultiMAE: Multi-modal Multi-task Masked Autoencoders Roman Bachmann David Mizrahi Andrei Atanov Amir Zamir 35 265 0 04 Apr 2022
BatchFormerV2: Exploring Sample Relationships for Dense Representation Learning Zhi Hou Baosheng Yu Chaoyue Wang Yibing Zhan Dacheng Tao ViT 29 11 0 04 Apr 2022
Dynamic Focus-aware Positional Queries for Semantic Segmentation Haoyu He Jianfei Cai Zizheng Pan Jing Liu Jing Zhang Dacheng Tao Bohan Zhuang 34 17 0 04 Apr 2022
Co-Teaching for Unsupervised Domain Adaptation and Expansion Kaibin Tian Qijie Wei Xirong Li 29 1 0 04 Apr 2022
Improving Vision Transformers by Revisiting High-frequency Components Jiawang Bai Liuliang Yuan Shutao Xia Shuicheng Yan Zhifeng Li Wei Liu ViT 16 90 0 03 Apr 2022
TransGeo: Transformer Is All You Need for Cross-view Image Geo-localization Sijie Zhu M. Shah Cheng Chen ViT 21 147 0 31 Mar 2022
Deformable Video Transformer Jue Wang Lorenzo Torresani ViT 24 28 0 31 Mar 2022
Surface Vision Transformers: Attention-Based Modelling applied to Cortical Analysis Simon Dahan Abdulah Fawaz Logan Z. J. Williams Chunhui Yang Timothy S. Coalson M. Glasser A. Edwards Daniel Rueckert E. C. Robinson MedIm ViT 40 20 0 30 Mar 2022
InstaFormer: Instance-Aware Image-to-Image Translation with Transformer Soohyun Kim Jongbeom Baek Jihye Park Gyeongnyeon Kim Seung Wook Kim ViT 39 47 0 30 Mar 2022