Training data-efficient image transformers & distillation through attention

23 December 2020

Alexandre Sablayrolles

Hervé Jégou

ViT

ArXiv PDF HTML

Papers citing "Training data-efficient image transformers & distillation through attention"

50 / 1,144 papers shown

Title
Dynamic Sparse Training versus Dense Training: The Unexpected Winner in Image Corruption Robustness Boqian Wu Q. Xiao Shunxin Wang N. Strisciuglio Mykola Pechenizkiy M. V. Keulen D. Mocanu Elena Mocanu OOD 3DH 52 0 0 03 Oct 2024
MAP: Unleashing Hybrid Mamba-Transformer Vision Backbone's Potential with Masked Autoregressive Pretraining Yunze Liu Li Yi Mamba 45 2 0 01 Oct 2024
Multi-View and Multi-Scale Alignment for Contrastive Language-Image Pre-training in Mammography Yuexi Du John Onofrey Nicha Dvornek VLM 50 1 0 26 Sep 2024
AI-driven View Guidance System in Intra-cardiac Echocardiography Imaging Jaeyoung Huh Paul Klein Gareth Funka-Lea Puneet Sharma A. Kapoor Young-Ho Kim 43 3 0 25 Sep 2024
OmniBench: Towards The Future of Universal Omni-Language Models Yizhi Li Ge Zhang Yinghao Ma Ruibin Yuan Kang Zhu ... Zhaoxiang Zhang Zachary Liu Emmanouil Benetos Wenhao Huang Chenghua Lin LRM 46 11 0 23 Sep 2024
Generalization in birdsong classification: impact of transfer learning methods and dataset characteristics Burooj Ghani Vincent J. Kalkman Bob Planqué Willem-Pier Vellinga L. Gill Dan Stowell VLM 32 5 0 21 Sep 2024
Multi-OCT-SelfNet: Integrating Self-Supervised Learning with Multi-Source Data Fusion for Enhanced Multi-Class Retinal Disease Classification Fatema Jannat Sina Gholami Jennifer I. Lim Theodore Leng Minhaj Nur Alam Hamed Tabkhi 28 0 0 17 Sep 2024
SkinMamba: A Precision Skin Lesion Segmentation Architecture with Cross-Scale Global State Modeling and Frequency Boundary Guidance Shun Zou Mingya Zhang Bingjian Fan Zhengyi Zhou Xiuguo Zou Mamba 29 3 0 17 Sep 2024
Frequency-Guided Masking for Enhanced Vision Self-Supervised Learning Amin Karimi Monsefi Mengxi Zhou Nastaran Karimi Monsefi Ser-Nam Lim Wei-Lun Chao R. Ramnath 44 1 0 16 Sep 2024
Effective Pre-Training of Audio Transformers for Sound Event Detection Florian Schmid T. Morocutti Francesco Foscarin Jan Schluter Paul Primus Gerhard Widmer ViT 25 2 0 14 Sep 2024
S-STE: Continuous Pruning Function for Efficient 2:4 Sparse Pre-training Yuezhou Hu Jun-Jie Zhu Jianfei Chen 36 0 0 13 Sep 2024
Token Turing Machines are Efficient Vision Models Purvish Jajal Nick Eliopoulos Benjamin Shiue-Hal Chou George K. Thiravathukal James C. Davis Yung-Hsiang Lu 93 0 0 11 Sep 2024
Brain-Inspired Stepwise Patch Merging for Vision Transformers Yonghao Yu Dongcheng Zhao Guobin Shen Yiting Dong Yi Zeng 50 0 0 11 Sep 2024
DetailCLIP: Detail-Oriented CLIP for Fine-Grained Tasks Amin Karimi Monsefi Kishore Prakash Sailaja Ali Alilooee Ser-Nam Lim R. Ramnath VLM 37 6 0 10 Sep 2024
Cross-attention Inspired Selective State Space Models for Target Sound Extraction Donghang Wu Yiwen Wang Xihong Wu T. Qu Mamba 32 3 0 07 Sep 2024
Learning to Discover Forgery Cues for Face Forgery Detection Jiahe Tian Peng-Wen Chen Cai Yu Xiaomeng Fu Xi Wang Jiao Dai Jizhong Han CVBM AAML 35 6 0 02 Sep 2024
A Survey of the Self Supervised Learning Mechanisms for Vision Transformers Asifullah Khan A. Sohail M. Fiaz Mehdi Hassan Tariq Habib Afridi ... Muhammad Zaigham Zaheer Kamran Ali Tangina Sultana Ziaurrehman Tanoli Naeem Akhter 45 3 0 30 Aug 2024
MePT: Multi-Representation Guided Prompt Tuning for Vision-Language Model Xinyang Wang Yi Yang Minfeng Zhu Kecheng Zheng Shi Liu Wei Chen VPVLM MLLM VLM 47 1 0 19 Aug 2024
FEDKIM: Adaptive Federated Knowledge Injection into Medical Foundation Models Xiaochen Wang Jiaqi Wang Houping Xiao J. Chen Fenglong Ma MedIm 63 7 0 17 Aug 2024
Breaking Class Barriers: Efficient Dataset Distillation via Inter-Class Feature Compensator Xin Zhang Jiawei Du Ping Liu Joey Tianyi Zhou DD 47 2 0 13 Aug 2024
Attacks and Defenses for Generative Diffusion Models: A Comprehensive Survey V. T. Truong Luan Ba Dang Long Bao Le DiffM MedIm 50 16 0 06 Aug 2024
Modelling Visual Semantics via Image Captioning to extract Enhanced Multi-Level Cross-Modal Semantic Incongruity Representation with Attention for Multimodal Sarcasm Detection Sajal Aggarwal Ananya Pandey Dinesh Kumar Vishwakarma 43 1 0 05 Aug 2024
Unsupervised Representation Learning by Balanced Self Attention Matching Daniel Shalam Simon Korman SSL 35 0 0 04 Aug 2024
DeMansia: Mamba Never Forgets Any Tokens Ricky Fang Mamba 19 0 0 04 Aug 2024
Scaling Backwards: Minimal Synthetic Pre-training? Ryo Nakamura Ryu Tadokoro Ryosuke Yamada Tim Puhlfürß Iro Laina Christian Rupprecht Walid Maalej Rio Yokota Hirokatsu Kataoka DD 19 2 0 01 Aug 2024
MimiQ: Low-Bit Data-Free Quantization of Vision Transformers with Encouraging Inter-Head Attention Similarity Kanghyun Choi Hyeyoon Lee Dain Kwon Sunjong Park Kyuyeun Kim Noseong Park Jinho Lee Jinho Lee MQ 46 1 0 29 Jul 2024
Depth-Wise Convolutions in Vision Transformers for Efficient Training on Small Datasets Tianxiao Zhang Wenju Xu Bo Luo Guanghui Wang ViT MDE 40 7 0 28 Jul 2024
Continual Distillation Learning: Knowledge Distillation in Prompt-based Continual Learning Qifan Zhang Yunhui Guo Yu Xiang VLM CLL 54 0 0 18 Jul 2024
PADRe: A Unifying Polynomial Attention Drop-in Replacement for Efficient Vision Transformer Pierre-David Létourneau Manish Kumar Singh Hsin-Pai Cheng Shizhong Han Yunxiao Shi Dalton Jones M. H. Langston Hong Cai Fatih Porikli 37 0 0 16 Jul 2024
Wicked Oddities: Selectively Poisoning for Effective Clean-Label Backdoor Attacks Quang H. Nguyen Nguyen Ngoc-Hieu The-Anh Ta Thanh Nguyen-Tang Kok-Seng Wong Hoang Thanh-Tung Khoa D. Doan AAML 33 2 0 15 Jul 2024
Background Adaptation with Residual Modeling for Exemplar-Free Class-Incremental Semantic Segmentation Anqi Zhang Guangyu Gao CLL VLM 38 4 0 13 Jul 2024
Neural-based Video Compression on Solar Dynamics Observatory Images Atefeh Khoshkhahtinat Ali Zafari P. Mehta Nasser M. Nasrabadi Barbara J. Thompson M. Kirk D. D. Silva 48 0 0 12 Jul 2024
On the Role of Discrete Tokenization in Visual Representation Learning Tianqi Du Yifei Wang Yisen Wang 49 7 0 12 Jul 2024
SUMix: Mixup with Semantic and Uncertain Information Huafeng Qin Xin Jin Hongyu Zhu Hongchao Liao M. El-Yacoubi Xinbo Gao UQCV 28 5 0 10 Jul 2024
MambaVision: A Hybrid Mamba-Transformer Vision Backbone Ali Hatamizadeh Jan Kautz Mamba 45 56 0 10 Jul 2024
Fish-Vista: A Multi-Purpose Dataset for Understanding & Identification of Traits from Images Kazi Sajeed Mehrab M. Maruf Arka Daw Harish Babu Manogaran Abhilash Neog ... Paula Mabee Wasila Dahdul Anuj Karpatne Wasila M Dahdul Anuj Karpatne 35 4 0 10 Jul 2024
HDKD: Hybrid Data-Efficient Knowledge Distillation Network for Medical Image Classification Omar S. El-Assiouti Ghada Hamed Dina Khattab H. M. Ebied 35 1 0 10 Jul 2024
Mobile Edge Intelligence for Large Language Models: A Contemporary Survey Guanqiao Qu Qiyuan Chen Wei Wei Zheng Lin Xianhao Chen Kaibin Huang 42 43 0 09 Jul 2024
HiT-SR: Hierarchical Transformer for Efficient Image Super-Resolution Xiang Zhang Yulun Zhang Fisher Yu 42 15 0 08 Jul 2024
Learning Motion Blur Robust Vision Transformers with Dynamic Early Exit for Real-Time UAV Tracking You Wu Xucheng Wang Dan Zeng Hengzhou Ye Xiaolan Xie Qijun Zhao Shuiwang Li 35 3 0 07 Jul 2024
Isomorphic Pruning for Vision Models Gongfan Fang Xinyin Ma Michael Bi Mi Xinchao Wang VLM ViT 34 6 0 05 Jul 2024
LPViT: Low-Power Semi-structured Pruning for Vision Transformers Kaixin Xu Zhe Wang Chunyun Chen Xue Geng Jie Lin Xulei Yang Min-man Wu Min Wu Xiaoli Li Weisi Lin ViT VLM 49 7 0 02 Jul 2024
A Sanity Check for AI-generated Image Detection Shilin Yan Ouxiang Li Jiayin Cai Y. Hao Xiaolong Jiang Yao Hu Weidi Xie VLM 66 20 0 27 Jun 2024
Mind the Graph When Balancing Data for Fairness or Robustness Jessica Schrouff Alexis Bellot Amal Rannen-Triki Alan Malek Isabela Albuquerque A. Gretton Alexander DÁmour Silvia Chiappa OOD CML 45 1 0 25 Jun 2024
LOGCAN++: Adaptive Local-global class-aware network for semantic segmentation of remote sensing imagery Xiaowen Ma Rongrong Lian Zhenkai Wu Hongbo Guo Mengting Ma Sensen Wu Zhenhong Du Siyang Song Wei Zhang 44 4 0 24 Jun 2024
Deciphering the Definition of Adversarial Robustness for post-hoc OOD Detectors Peter Lorenz Mario Fernandez Jens Müller Ullrich Kothe AAML 78 1 0 21 Jun 2024
CT3D++: Improving 3D Object Detection with Keypoint-induced Channel-wise Transformer Hualian Sheng Sijia Cai Na Zhao Bing Deng Qiao Liang Min-Jian Zhao Jieping Ye 3DPC 42 0 0 12 Jun 2024
Adaptively Bypassing Vision Transformer Blocks for Efficient Visual Tracking Xiangyang Yang Dan Zeng Xucheng Wang You Wu Hengzhou Ye Qijun Zhao Shuiwang Li 59 3 0 12 Jun 2024
Particle Multi-Axis Transformer for Jet Tagging Muhammad Usman M. Shahid Maheen Ejaz Ummay Hani Nayab Fatima Abdul Rehman Khan Asifullah Khan Nasir Majid Mirza 33 3 0 09 Jun 2024
DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data Qihao Liu Yi Zhang Song Bai Adam Kortylewski Alan Yuille 36 9 0 06 Jun 2024