Training data-efficient image transformers & distillation through attention

23 December 2020

Alexandre Sablayrolles

Hervé Jégou

ViT

ArXiv PDF HTML

Papers citing "Training data-efficient image transformers & distillation through attention"

50 / 1,158 papers shown

Title
Attention-aware Semantic Communications for Collaborative Inference Jiwoong Im Nayoung Kwon Taewoo Park Jiheon Woo Jaeho Lee Yongjune Kim 46 2 0 23 Feb 2024
Beyond Uniform Scaling: Exploring Depth Heterogeneity in Neural Architectures Akash Guna R.T Arnav Chavan Deepak Gupta MDE 32 0 0 19 Feb 2024
FViT: A Focal Vision Transformer with Gabor Filter Yulong Shi Mingwei Sun Yongshuai Wang Rui Wang 55 4 0 17 Feb 2024
Learning Low-Rank Feature for Thorax Disease Classification Rajeev Goel Utkarsh Nath Yancheng Wang Alvin C. Silva Teresa Wu Yingzhen Yang 22 0 0 14 Feb 2024
TransAxx: Efficient Transformers with Approximate Computing Dimitrios Danopoulos Georgios Zervakis Dimitrios Soudris Jörg Henkel ViT 42 2 0 12 Feb 2024
Text-to-Code Generation with Modality-relative Pre-training Fenia Christopoulou Guchun Zhang Gerasimos Lampouras AI4TS 18 1 0 08 Feb 2024
RepQuant: Towards Accurate Post-Training Quantization of Large Transformer Models via Scale Reparameterization Zhikai Li Xuewen Liu Jing Zhang Qingyi Gu MQ 37 7 0 08 Feb 2024
Implicit Bias and Fast Convergence Rates for Self-attention Bhavya Vasudeva Puneesh Deora Christos Thrampoulidis 34 13 0 08 Feb 2024
CoBra: Complementary Branch Fusing Class and Semantic Knowledge for Robust Weakly Supervised Semantic Segmentation Woojung Han Seil Kang Kyobin Choo Seong Jae Hwang 19 0 0 05 Feb 2024
DeSparsify: Adversarial Attack Against Token Sparsification Mechanisms in Vision Transformers Oryan Yehezkel Alon Zolfi Amit Baras Yuval Elovici A. Shabtai AAML 29 0 0 04 Feb 2024
Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey Yi Xin Jianjiang Yang Haodi Zhou Junlong Du Junlong Du Yue Fan Qing Li Qing Li Yuntao Du VLM 73 75 0 03 Feb 2024
CascadedGaze: Efficiency in Global Context Extraction for Image Restoration Amirhosein Ghasemabadi Muhammad Kamran Janjua Mohammad Salameh Chunhua Zhou Fengyu Sun Di Niu 35 11 0 26 Jan 2024
MPTQ-ViT: Mixed-Precision Post-Training Quantization for Vision Transformer Y. Tai An-Yeu Wu Wu MQ 28 6 0 26 Jan 2024
Producing Plankton Classifiers that are Robust to Dataset Shift Cheng Chen S. Kyathanahally Marta Reyes Stefanie Merkli E. Merz Emanuele Francazi Marvin Hoege F. Pomati M. Baity-Jesi 16 2 0 25 Jan 2024
Sparse and Transferable Universal Singular Vectors Attack Kseniia Kuvshinova Olga Tsymboi Ivan V. Oseledets AAML 35 0 0 25 Jan 2024
OnDev-LCT: On-Device Lightweight Convolutional Transformers towards federated learning Chu Myaet Thwal Minh N. H. Nguyen Ye Lin Tun Seongjin Kim My T. Thai Choong Seon Hong 61 5 0 22 Jan 2024
Transformer-CNN Fused Architecture for Enhanced Skin Lesion Segmentation Siddharth Tiwari MedIm ViT 42 0 0 10 Jan 2024
Setting the Record Straight on Transformer Oversmoothing G. Dovonon M. Bronstein Matt J. Kusner 22 5 0 09 Jan 2024
Morphing Tokens Draw Strong Masked Image Models Taekyung Kim Byeongho Heo Dongyoon Han 49 3 0 30 Dec 2023
An Empirical Study of Scaling Law for OCR Miao Rang Zhenni Bi Chuanjian Liu Yunhe Wang Kai Han 33 6 0 29 Dec 2023
Adaptive Depth Networks with Skippable Sub-Paths Woochul Kang 28 1 0 27 Dec 2023
Video Recognition in Portrait Mode Mingfei Han Linjie Yang Xiaojie Jin Jiashi Feng Xiaojun Chang Heng Wang 30 3 0 21 Dec 2023
Polynomial-based Self-Attention for Table Representation learning Jayoung Kim Yehjin Shin Jeongwhan Choi Hyowon Wi Noseong Park LMTD 21 2 0 12 Dec 2023
MaxQ: Multi-Axis Query for N:M Sparsity Network Jingyang Xiang Siqi Li Junhao Chen Zhuangzhi Chen Tianxin Huang Linpeng Peng Yong-Jin Liu 16 0 0 12 Dec 2023
MIMIR: Masked Image Modeling for Mutual Information-based Adversarial Robustness Xiaoyun Xu Shujian Yu Jingzheng Wu S. Picek AAML 35 0 0 08 Dec 2023
Graph Convolutions Enrich the Self-Attention in Transformers! Jeongwhan Choi Hyowon Wi Jayoung Kim Yehjin Shin Kookjin Lee Nathaniel Trask Noseong Park 25 4 0 07 Dec 2023
Diversify, Don't Fine-Tune: Scaling Up Visual Recognition Training with Synthetic Images Zhuoran Yu Chenchen Zhu Sean Culatana Raghuraman Krishnamoorthi Fanyi Xiao Yong Jae Lee 117 14 0 04 Dec 2023
Improve Supervised Representation Learning with Masked Image Modeling Kaifeng Chen Daniel M. Salz Huiwen Chang Kihyuk Sohn Dilip Krishnan Mojtaba Seyedhosseini SSL ViT 37 3 0 01 Dec 2023
EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything Yunyang Xiong Bala Varadarajan Lemeng Wu Xiaoyu Xiang Fanyi Xiao ... Dilin Wang Fei Sun Forrest N. Iandola Raghuraman Krishnamoorthi Vikas Chandra VLM 40 139 0 01 Dec 2023
SCHEME: Scalable Channel Mixer for Vision Transformers Deepak Sridhar Yunsheng Li Nuno Vasconcelos 44 0 0 01 Dec 2023
Initializing Models with Larger Ones Zhiqiu Xu Yanjie Chen Kirill Vishniakov Yida Yin Zhiqiang Shen Trevor Darrell Lingjie Liu Zhuang Liu 30 17 0 30 Nov 2023
Improving Interpretation Faithfulness for Vision Transformers Lijie Hu Yixin Liu Ninghao Liu Mengdi Huai Lichao Sun Di Wang 32 5 0 29 Nov 2023
Improving Feature Stability during Upsampling -- Spectral Artifacts and the Importance of Spatial Context Shashank Agnihotri Julia Grabinski M. Keuper 30 6 0 29 Nov 2023
PViT-6D: Overclocking Vision Transformers for 6D Pose Estimation with Confidence-Level Prediction and Pose Tokens Sebastian Stapf Tobias Bauernfeind Marco Riboldi ViT 25 1 0 29 Nov 2023
QuadraNet: Improving High-Order Neural Interaction Efficiency with Hardware-Aware Quadratic Neural Networks Chenhui Xu Fuxun Yu Zirui Xu Chenchen Liu Jinjun Xiong Xiang Chen 33 4 0 29 Nov 2023
Efficient Stitchable Task Adaptation Haoyu He Zizheng Pan Jing Liu Jianfei Cai Bohan Zhuang 26 3 0 29 Nov 2023
No Representation Rules Them All in Category Discovery S. Vaze Andrea Vedaldi Andrew Zisserman OOD 37 31 0 28 Nov 2023
Vulnerability Analysis of Transformer-based Optical Character Recognition to Adversarial Attacks Lucas Beerens D. Higham 36 1 0 28 Nov 2023
HEViTPose: High-Efficiency Vision Transformer for Human Pose Estimation Chengpeng Wu Guangxing Tan Chunyu Li ViT 21 0 0 22 Nov 2023
Improving Source-Free Target Adaptation with Vision Transformers Leveraging Domain Representation Images Gauransh Sawhney Daksh Dave Adeel Ahmed Jiechao Gao Khalid Saleem 23 0 0 21 Nov 2023
Pursing the Sparse Limitation of Spiking Deep Learning Structures Hao-Ran Cheng Jiahang Cao Erjia Xiao Mengshu Sun Le Yang Jize Zhang Xue Lin B. Kailkhura Kaidi Xu Renjing Xu 16 1 0 18 Nov 2023
Improved TokenPose with Sparsity Anning Li ViT 34 0 0 16 Nov 2023
Contrastive Transformer Learning with Proximity Data Generation for Text-Based Person Search Hefeng Wu Weifeng Chen Zhibin Liu Tianshui Chen Zhiguang Chen Liang Lin 28 11 0 15 Nov 2023
4K-Resolution Photo Exposure Correction at 125 FPS with ~8K Parameters Yijie Zhou Chao Li Jin Liang Tianyi Xu Xin Liu Jun Xu 3DV 21 10 0 15 Nov 2023
Learning Contrastive Self-Distillation for Ultra-Fine-Grained Visual Categorization Targeting Limited Samples Ziye Fang Xin Jiang Hao Tang Zechao Li 30 12 0 10 Nov 2023
Dynamic Association Learning of Self-Attention and Convolution in Image Restoration Kui Jiang Xuemei Jia Wenxin Huang Wenbin Wang Zheng Wang Junjun Jiang 20 1 0 09 Nov 2023
Rotation Invariant Transformer for Recognizing Object in UAVs Shuo Chen Mang Ye Bo Du ViT 32 18 0 05 Nov 2023
BarcodeBERT: Transformers for Biodiversity Analysis Pablo Millán Arias Niousha Sadjadi Monireh Safari ZeMing Gong Austin T. Wang ... Iuliia Zarubiieva Dirk Steinke Lila Kari Angel X. Chang Graham W. Taylor 42 7 0 04 Nov 2023
What Makes Pre-Trained Visual Representations Successful for Robust Manipulation? Kaylee Burns Zach Witzel Jubayer Ibn Hamid Tianhe Yu Chelsea Finn Karol Hausman OOD SSL 27 22 0 03 Nov 2023
PAUMER: Patch Pausing Transformer for Semantic Segmentation Evann Courdier Prabhu Teja Sivaprasad F. Fleuret 37 2 0 01 Nov 2023