Training data-efficient image transformers & distillation through attention

23 December 2020

Alexandre Sablayrolles

Hervé Jégou

ViT

ArXiv PDF HTML

Papers citing "Training data-efficient image transformers & distillation through attention"

50 / 1,158 papers shown

Title
Mixture-of-Experts for Open Set Domain Adaptation: A Dual-Space Detection Approach Zhenbang Du Jiayu An Yunlu Tu Jiahao Hong Dongrui Wu MoE 20 1 0 01 Nov 2023
OpenForest: A data catalogue for machine learning in forest monitoring Arthur Ouaknine T. Kattenborn Etienne Laliberté David Rolnick 43 5 0 01 Nov 2023
SolarFormer: Multi-scale Transformer for Solar PV Profiling Adrian de Luis Minh-Triet Tran Taisei Hanyu Anh Tran Haitao Liao Roy McCann Alan Mantooth Ying Huang Ngan Le 33 3 0 30 Oct 2023
Are Natural Domain Foundation Models Useful for Medical Image Classification? Joana Palés Huix Adithya Raju Ganeshan Johan Fredin Haslum Magnus P Soderberg Christos Matsoukas Kevin Smith OOD MedIm VLM 24 30 0 30 Oct 2023
TransXNet: Learning Both Global and Local Dynamics with a Dual Dynamic Token Mixer for Visual Recognition Meng Lou Hong-Yu Zhou Sibei Yang Yizhou Yu Chuan Wu Yizhou Yu ViT 44 36 0 30 Oct 2023
Towards Generalized Multi-stage Clustering: Multi-view Self-distillation Jiatai Wang Zhiwei Xu Xin Wang Tao Li 11 1 0 29 Oct 2023
Understanding the Effects of Projectors in Knowledge Distillation Yudong Chen Sen Wang Jiajun Liu Xuwei Xu Frank de Hoog Brano Kusy Zi Huang 26 0 0 26 Oct 2023
Gramian Attention Heads are Strong yet Efficient Vision Learners Jongbin Ryu Dongyoon Han J. Lim 30 1 0 25 Oct 2023
Auxiliary Features-Guided Super Resolution for Monte Carlo Rendering Qiqi Hou Feng Liu SupR 17 4 0 20 Oct 2023
Minimalist and High-Performance Semantic Segmentation with Plain Vision Transformers Yuanduo Hong Jue Wang Weichao Sun Huihui Pan VLM ViT 37 7 0 19 Oct 2023
Medical Image Segmentation via Sparse Coding Decoder Long Zeng Kaigui Wu MedIm 26 3 0 17 Oct 2023
Multi-domain improves out-of-distribution and data-limited scenarios for medical image analysis Ece Ozkan Xavier Boix OOD 28 0 0 10 Oct 2023
EViT: An Eagle Vision Transformer with Bi-Fovea Self-Attention Yulong Shi Mingwei Sun Yongshuai Wang Hui Sun Zengqiang Chen 34 4 0 10 Oct 2023
Conformal Prediction for Deep Classifier via Label Ranking Jianguo Huang Huajun Xi Linjun Zhang Huaxiu Yao Yue Qiu Hongxin Wei 39 21 0 10 Oct 2023
CoinSeg: Contrast Inter- and Intra- Class Representations for Incremental Segmentation Zekang Zhang Guangyu Gao Jianbo Jiao C. Liu Yunchao Wei 48 21 0 10 Oct 2023
Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity Lu Yin You Wu Zhenyu (Allen) Zhang Cheng-Yu Hsieh Yaqing Wang ... Mykola Pechenizkiy Yi Liang Michael Bendersky Zhangyang Wang Shiwei Liu 28 78 0 08 Oct 2023
Low-Resolution Self-Attention for Semantic Segmentation Yu-Huan Wu Shi-Chen Zhang Yun-Hai Liu Le Zhang Xin Zhan Daquan Zhou Jiashi Feng Ming-Ming Cheng Liangli Zhen ViT 45 3 0 08 Oct 2023
NOLA: Compressing LoRA using Linear Combination of Random Basis Soroush Abbasi Koohpayegani K. Navaneet Parsa Nooralinejad Soheil Kolouri Hamed Pirsiavash 40 12 0 04 Oct 2023
Efficient Supervised Training of Audio Transformers for Music Representation Learning Pablo Alonso-Jiménez Xavier Serra Dmitry Bogdanov ViT 35 3 0 28 Sep 2023
Species196: A One-Million Semi-supervised Dataset for Fine-grained Species Recognition W. He Kai Han Ying Nie Chengcheng Wang Yunhe Wang VLM 45 6 0 25 Sep 2023
Weight Averaging Improves Knowledge Distillation under Domain Shift Valeriy Berezovskiy Nikita Morozov MoMe 24 1 0 20 Sep 2023
Interpretability-Aware Vision Transformer Yao Qiang Chengyin Li Prashant Khanduri D. Zhu ViT 82 7 0 14 Sep 2023
3D Transformer based on deformable patch location for differential diagnosis between Alzheimer's disease and Frontotemporal dementia H. Nguyen Michael Clement Boris Mansencal Pierrick Coupé MedIm 28 0 0 06 Sep 2023
Prototype-based Dataset Comparison N. V. Noord 31 6 0 05 Sep 2023
AGS: An Dataset and Taxonomy for Domestic Scene Sound Event Recognition Nan Che Chenrui Liu Fei Yu 33 0 0 30 Aug 2023
Uncovering the Hidden Cost of Model Compression Diganta Misra Muawiz Chaudhary Agam Goyal Bharat Runwal Pin-Yu Chen VLM 33 0 0 29 Aug 2023
PanoSwin: a Pano-style Swin Transformer for Panorama Understanding Zhixin Ling Zhen Xing Xiangdong Zhou Manliang Cao G. Zhou ViT 26 17 0 28 Aug 2023
With a Little Help from your own Past: Prototypical Memory Networks for Image Captioning Manuele Barraco Sara Sarto Marcella Cornia Lorenzo Baraldi Rita Cucchiara VLM 55 19 0 23 Aug 2023
TurboViT: Generating Fast Vision Transformers via Generative Architecture Search Alexander Wong Saad Abbasi Saeejith Nair ViT 27 1 0 22 Aug 2023
UnLoc: A Unified Framework for Video Localization Tasks Shengjia Yan Xuehan Xiong Arsha Nagrani Anurag Arnab Zhonghao Wang Weina Ge David A. Ross Cordelia Schmid 31 53 0 21 Aug 2023
MGMAE: Motion Guided Masking for Video Masked Autoencoding Bingkun Huang Zhiyu Zhao Guozhen Zhang Yu Qiao Limin Wang 39 30 0 21 Aug 2023
CoNe: Contrast Your Neighbours for Supervised Image Classification Mingkai Zheng Shan You Lang Huang Xiu Su Fei Wang Chao Qian Xiaogang Wang Chang Xu VLM 26 0 0 21 Aug 2023
HODN: Disentangling Human-Object Feature for HOI Detection Shuman Fang Zhiwen Lin Ke Yan Jie Li Xianming Lin Rongrong Ji 48 5 0 20 Aug 2023
Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformers Tobias Christian Nauen Sebastián M. Palacio Federico Raue Andreas Dengel 42 3 0 18 Aug 2023
AudioFormer: Audio Transformer learns audio feature representations from discrete acoustic codes Zhaohui Li Haitao Wang Xinghua Jiang 40 1 0 14 Aug 2023
Advancing Natural-Language Based Audio Retrieval with PaSST and Large Audio-Caption Data Sets Paul Primus Khaled Koutini Gerhard Widmer 19 13 0 08 Aug 2023
SSTFormer: Bridging Spiking Neural Network and Memory Support Transformer for Frame-Event based Recognition Xiao Wang Zong-Yao Wu Yao Rong Lin Zhu Bowei Jiang Jin Tang Yonghong Tian ViT 71 15 0 08 Aug 2023
FLIQS: One-Shot Mixed-Precision Floating-Point and Integer Quantization Search Jordan Dotzel Gang Wu Andrew Li M. Umar Yun Ni ... Liqun Cheng Martin G. Dixon N. Jouppi Quoc V. Le Sheng R. Li MQ 27 3 0 07 Aug 2023
DETR Doesn't Need Multi-Scale or Locality Design Yutong Lin Yuhui Yuan Zheng-Wei Zhang Chen Li Nanning Zheng Han Hu 37 5 0 03 Aug 2023
Deep Learning and Computer Vision for Glaucoma Detection: A Review Mona Ashtari-Majlan Mohammad Mahdi Dehshibi David Masip 32 9 0 31 Jul 2023
BARTPhoBEiT: Pre-trained Sequence-to-Sequence and Image Transformers Models for Vietnamese Visual Question Answering Khiem Vinh Tran Kiet Van Nguyen N. Nguyen ViT 23 2 0 28 Jul 2023
Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models Dong Lu Zhiqiang Wang Teng Wang Weili Guan Hongchang Gao Feng Zheng AAML 51 65 0 26 Jul 2023
Visual Prompt Flexible-Modal Face Anti-Spoofing Zitong Yu Rizhao Cai Yawen Cui Ajian Liu Changsheng Chen 36 6 0 26 Jul 2023
Foundational Models Defining a New Era in Vision: A Survey and Outlook Muhammad Awais Muzammal Naseer Salman Khan Rao Muhammad Anwer Hisham Cholakkal M. Shah Ming Yang F. Khan VLM 38 118 0 25 Jul 2023
Quantized Feature Distillation for Network Quantization Kevin Zhu Yin He Jianxin Wu MQ 29 9 0 20 Jul 2023
Improving Domain Generalization for Sound Classification with Sparse Frequency-Regularized Transformer Honglin Mu Wentian Xia Wanxiang Che 22 1 0 19 Jul 2023
Hierarchical Spatiotemporal Transformers for Video Object Segmentation Jun-Sang Yoo H. Lee Seung‐Won Jung VOS 29 1 0 17 Jul 2023
Mitigating Adversarial Vulnerability through Causal Parameter Estimation by Adversarial Double Machine Learning Byung-Kwan Lee Junho Kim Yonghyun Ro AAML 18 9 0 14 Jul 2023
Random Position Adversarial Patch for Vision Transformers Mingzhen Shao ViT AAML 35 2 0 09 Jul 2023
All in One: Exploring Unified Vision-Language Tracking with Multi-Modal Alignment Chunhui Zhang Xin Sun Li Liu Yiqian Yang Qiong Liu Xiaoping Zhou Yanfeng Wang 40 15 0 07 Jul 2023