Training data-efficient image transformers & distillation through attention

23 December 2020

Alexandre Sablayrolles

Hervé Jégou

ViT

ArXiv PDF HTML

Papers citing "Training data-efficient image transformers & distillation through attention"

50 / 1,319 papers shown

Title
Transformer-CNN Cohort: Semi-supervised Semantic Segmentation by the Best of Both Students Xueye Zheng Yuan Luo Hao Wang Chong Fu Lin Wang ViT 41 18 0 06 Sep 2022
An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modeling Tsu-jui Fu Linjie Li Zhe Gan Kevin Qinghong Lin William Yang Wang Lijuan Wang Zicheng Liu VLM 26 64 0 04 Sep 2022
TokenCut: Segmenting Objects in Images and Videos with Self-supervised Transformer and Normalized Cut Yangtao Wang Xiaoke Shen Yuan. Yuan Yuming Du Maomao Li S. Hu James L. Crowley Dominique Vaufreydaz VOS ViT 27 78 0 01 Sep 2022
MRL: Learning to Mix with Attention and Convolutions Shlok Mohta Hisahiro Suganuma Yoshiki Tanaka 28 2 0 30 Aug 2022
SB-SSL: Slice-Based Self-Supervised Transformers for Knee Abnormality Classification from MRI Sara Atito Syed Muhammad Anwar Muhammad Awais Josef Kitler ViT MedIm 29 12 0 29 Aug 2022
An Access Control Method with Secret Key for Semantic Segmentation Models Teru Nagamori Ryota Iijima Hitoshi Kiya 24 0 0 28 Aug 2022
MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining Xiaoyi Dong Jianmin Bao Yinglin Zheng Ting Zhang Dongdong Chen ... Weiming Zhang Lu Yuan Dong Chen Fang Wen Nenghai Yu CLIP VLM 54 158 0 25 Aug 2022
Masked Autoencoders Enable Efficient Knowledge Distillers Yutong Bai Zeyu Wang Junfei Xiao Chen Wei Huiyu Wang Alan Yuille Yuyin Zhou Cihang Xie CLL 32 39 0 25 Aug 2022
Improved Zero-Shot Audio Tagging & Classification with Patchout Spectrogram Transformers Paul Primus Gerhard Widmer VLM 21 5 0 24 Aug 2022
Federated Self-Supervised Contrastive Learning and Masked Autoencoder for Dermatological Disease Diagnosis Yawen Wu Dewen Zeng Zhepeng Wang Yi Sheng Lei Yang A. James Yiyu Shi Jingtong Hu 20 7 0 24 Aug 2022
Efficient Attention-free Video Shift Transformers Adrian Bulat Brais Martínez Georgios Tzimiropoulos ViT 29 1 0 23 Aug 2022
How good are deep models in understanding the generated images? Ali Borji OOD 24 6 0 23 Aug 2022
ProtoPFormer: Concentrating on Prototypical Parts in Vision Transformers for Interpretable Image Recognition Mengqi Xue Qihan Huang Haofei Zhang Lechao Cheng Mingli Song Ming-hui Wu Mingli Song ViT 27 52 0 22 Aug 2022
A Unified Analysis of Mixed Sample Data Augmentation: A Loss Function Perspective Chanwoo Park Sangdoo Yun Sanghyuk Chun AAML 21 32 0 21 Aug 2022
A Multi-Head Model for Continual Learning via Out-of-Distribution Replay Gyuhak Kim Zixuan Ke Bin Liu VLM CLL OODD 25 29 0 20 Aug 2022
Exploring Adversarial Robustness of Vision Transformers in the Spectral Perspective Gihyun Kim Juyeop Kim Jong-Seok Lee AAML ViT 24 4 0 20 Aug 2022
Accelerating Vision Transformer Training via a Patch Sampling Schedule Bradley McDanel C. Huynh ViT 30 1 0 19 Aug 2022
Improved Image Classification with Token Fusion Keong-Hun Choi Jin-Woo Kim Yaolong Wang J. Ha ViT 19 0 0 19 Aug 2022
GSRFormer: Grounded Situation Recognition Transformer with Alternate Semantic Attention Refinement Zhi-Qi Cheng Qianwen Dai Siyao Li Teruko Mitamura Alexander G. Hauptmann 16 34 0 18 Aug 2022
Your ViT is Secretly a Hybrid Discriminative-Generative Diffusion Model Xiulong Yang Sheng-Min Shih Yinlin Fu Xiaoting Zhao Shihao Ji DiffM 33 56 0 16 Aug 2022
The LAM Dataset: A Novel Benchmark for Line-Level Handwritten Text Recognition S. Cascianelli Vittorio Pippi Martin Maarand Marcella Cornia Lorenzo Baraldi Christopher Kermorvant Rita Cucchiara 21 7 0 16 Aug 2022
USB: A Unified Semi-supervised Learning Benchmark for Classification Yidong Wang Hao Chen Yue Fan Wangbin Sun R. Tao ... T. Shinozaki Bernt Schiele Jindong Wang Xingxu Xie Yue Zhang 27 113 0 12 Aug 2022
How Well Do Vision Transformers (VTs) Transfer To The Non-Natural Image Domain? An Empirical Study Involving Art Classification Vincent Tonkes M. Sabatelli ViT 27 6 0 09 Aug 2022
GRIT-VLP: Grouped Mini-batch Sampling for Efficient Vision and Language Pre-training Jaeseok Byun Taebaek Hwang Jianlong Fu Taesup Moon VLM 23 11 0 08 Aug 2022
TransMatting: Enhancing Transparent Objects Matting with Transformers Huanqia Cai Fanglei Xue Lele Xu Lili Guo ViT 11 20 0 05 Aug 2022
Transformers as Meta-Learners for Implicit Neural Representations Yinbo Chen Xiaolong Wang AI4CE 24 60 0 04 Aug 2022
Multi-Feature Vision Transformer via Self-Supervised Representation Learning for Improvement of COVID-19 Diagnosis Xiao Qi D. Foran J. Nosher I. Hacihaliloglu ViT MedIm 30 3 0 03 Aug 2022
Making the Best of Both Worlds: A Domain-Oriented Transformer for Unsupervised Domain Adaptation Wen-hui Ma Jinming Zhang Shuang Li Chi Harold Liu Yulin Wang Wei Li 26 14 0 02 Aug 2022
Pose Uncertainty Aware Movement Synchrony Estimation via Spatial-Temporal Graph Transformer Jicheng Li Anjana Bhat R. Barmaki ViT 27 5 0 01 Aug 2022
Understanding Adversarial Robustness of Vision Transformers via Cauchy Problem Zheng Wang Wenjie Ruan ViT 42 8 0 01 Aug 2022
Cross Attention Based Style Distribution for Controllable Person Image Synthesis Xinyue Zhou M. Yin Xinyuan Chen Li Sun Changxin Gao Qingli Li DiffM 14 54 0 01 Aug 2022
Local Perception-Aware Transformer for Aerial Tracking Changhong Fu Wei Peng Sihang Li Junjie Ye Ziang Cao 36 8 0 01 Aug 2022
Momentum Transformer: Closing the Performance Gap Between Self-attention and Its Linearization T. Nguyen Richard G. Baraniuk Robert M. Kirby Stanley J. Osher Bao Wang 32 9 0 01 Aug 2022
UAVM: Towards Unifying Audio and Visual Models Yuan Gong Alexander H. Liu Andrew Rouditchenko James R. Glass 30 21 0 29 Jul 2022
Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer Hao Shao Letian Wang Ruobing Chen Hongsheng Li Y. Liu 47 195 0 28 Jul 2022
Jigsaw-ViT: Learning Jigsaw Puzzles in Vision Transformer Yingyi Chen Xiaoke Shen Yahui Liu Qinghua Tao Johan A. K. Suykens AAML ViT 28 22 0 25 Jul 2022
Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation Jiaming Zhang Kailun Yang Haowen Shi Simon Reiß Kunyu Peng Chaoxiang Ma Haodong Fu Philip H. S. Torr Kaiwei Wang Rainer Stiefelhagen ViT MDE 31 36 0 25 Jul 2022
MAR: Masked Autoencoders for Efficient Action Recognition Zhiwu Qing Shiwei Zhang Ziyuan Huang Xiang Wang Yuehuang Wang Yiliang Lv Changxin Gao Nong Sang 32 42 0 24 Jul 2022
High-Resolution Swin Transformer for Automatic Medical Image Segmentation Chen Wei Shenghan Ren Kaitai Guo Haihong Hu Jimin Liang ViT OOD MedIm 25 36 0 23 Jul 2022
An Impartial Take to the CNN vs Transformer Robustness Contest Francesco Pinto Philip Torr P. Dokania UQCV AAML 33 49 0 22 Jul 2022
PanGu-Coder: Program Synthesis with Function-Level Language Modeling Fenia Christopoulou Gerasimos Lampouras Milan Gritta Guchun Zhang Yinpeng Guo ... Guangtai Liang Jia Wei Xin Jiang Qianxiang Wang Qun Liu ELM SyDa ALM 45 74 0 22 Jul 2022
Emotion Separation and Recognition from a Facial Expression by Generating the Poker Face with Vision Transformers Jia Li Jian‐Hui Nie Dan Guo Richang Hong Meng Wang ViT 32 14 0 22 Jul 2022
Geodesic-Former: a Geodesic-Guided Few-shot 3D Point Cloud Instance Segmenter T. Ngo Khoi Duc Minh Nguyen 3DPC 19 4 0 22 Jul 2022
Exploring Fine-Grained Audiovisual Categorization with the SSW60 Dataset Grant Van Horn Rui Qian Kimberly Wilber Hartwig Adam Oisin Mac Aodha Serge Belongie 27 10 0 21 Jul 2022
Towards Efficient Adversarial Training on Vision Transformers Boxi Wu Jindong Gu Zhifeng Li Deng Cai Xiaofei He Wei Liu ViT AAML 46 38 0 21 Jul 2022
Locality Guidance for Improving Vision Transformers on Tiny Datasets Kehan Li Runyi Yu Zhennan Wang Li-ming Yuan Guoli Song Jie Chen ViT 32 44 0 20 Jul 2022
Vision Transformers: From Semantic Segmentation to Dense Prediction Li Zhang Jiachen Lu Sixiao Zheng Xinxuan Zhao Xiatian Zhu Yanwei Fu Tao Xiang Jianfeng Feng Philip H. S. Torr ViT 27 7 0 19 Jul 2022
Assaying Out-Of-Distribution Generalization in Transfer Learning F. Wenzel Andrea Dittadi Peter V. Gehler Carl-Johann Simon-Gabriel Max Horn ... Chris Russell Thomas Brox Bernt Schiele Bernhard Schölkopf Francesco Locatello OOD OODD AAML 60 71 0 19 Jul 2022
GAFX: A General Audio Feature eXtractor Zhaoyang Bu Han Zhang Xiaohu Zhu 30 0 0 19 Jul 2022
Time Is MattEr: Temporal Self-supervision for Video Transformers Sukmin Yun Jaehyung Kim Dongyoon Han Hwanjun Song Jung-Woo Ha Jinwoo Shin ViT 19 12 0 19 Jul 2022