Training data-efficient image transformers & distillation through attention

23 December 2020

Alexandre Sablayrolles

Hervé Jégou

ViT

ArXiv PDF HTML

Papers citing "Training data-efficient image transformers & distillation through attention"

50 / 1,144 papers shown

Title
DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data Qihao Liu Yi Zhang Song Bai Adam Kortylewski Alan Yuille 36 9 0 06 Jun 2024
ReDistill: Residual Encoded Distillation for Peak Memory Reduction of CNNs Fang Chen Gourav Datta Mujahid Al Rafi Hyeran Jeon Meng Tang 91 1 0 06 Jun 2024
The 3D-PC: a benchmark for visual perspective taking in humans and machines Drew Linsley Peisen Zhou A. Ashok Akash Nagaraj Gaurav Gaonkar Francis E Lewis Zygmunt Pizlo Thomas Serre 48 6 0 06 Jun 2024
Language-guided Detection and Mitigation of Unknown Dataset Bias Zaiying Zhao Soichiro Kumano Toshihiko Yamasaki 36 2 0 05 Jun 2024
Tiny models from tiny data: Textual and null-text inversion for few-shot distillation Erik Landolsi Fredrik Kahl DiffM 58 1 0 05 Jun 2024
GrootVL: Tree Topology is All You Need in State Space Model Yicheng Xiao Lin Song Shaoli Huang Jiangshan Wang Siyu Song Yixiao Ge Xiu Li Ying Shan Mamba 41 10 0 04 Jun 2024
Scaling White-Box Transformers for Vision Jinrui Yang Xianhang Li Druv Pai Yuyin Zhou Yi-An Ma Yaodong Yu Cihang Xie ViT 44 9 0 30 May 2024
Sharing Key Semantics in Transformer Makes Efficient Image Restoration Bin Ren Yawei Li Jingyun Liang Rakesh Ranjan Mengyuan Liu Rita Cucchiara Luc Van Gool Ming-Hsuan Yang N. Sebe 37 3 0 30 May 2024
P $^2$ -ViT: Power-of-Two Post-Training Quantization and Acceleration for Fully Quantized Vision Transformer Huihong Shi Xin Cheng Wendong Mao Zhongfeng Wang MQ 40 3 0 30 May 2024
Don't drop your samples! Coherence-aware training benefits Conditional diffusion Nicolas Dufour Victor Besnier Vicky Kalogeiton David Picard DiffM 51 2 0 30 May 2024
Instruct-ReID++: Towards Universal Purpose Instruction-Guided Person Re-identification Weizhen He Yiheng Deng Yunfeng Yan Feng Zhu Yizhou Wang Lei Bai Qingsong Xie Donglian Qi Wanli Ouyang Shixiang Tang 95 2 0 28 May 2024
Exploiting the Layered Intrinsic Dimensionality of Deep Models for Practical Adversarial Training Enes Altinisik Safa Messaoud H. Sencar Hassan Sajjad Sanjay Chawla AAML 48 0 0 27 May 2024
Understanding the Effect of using Semantically Meaningful Tokens for Visual Representation Learning N. Kalibhat Priyatham Kattakinda Arman Zarei Nikita Seleznev Sam Sharpe Senthil Kumar S. Feizi ViT 34 0 0 26 May 2024
Accelerating Transformers with Spectrum-Preserving Token Merging Hoai-Chau Tran D. M. Nguyen Duy M. Nguyen Trung Thanh Nguyen Ngan Le Pengtao Xie Daniel Sonntag James Y. Zou Binh T. Nguyen Mathias Niepert 36 8 0 25 May 2024
Dinomaly: The Less Is More Philosophy in Multi-Class Unsupervised Anomaly Detection Jia Guo Shuai Lu Weihang Zhang Huiqi Li Huiqi Li Hongen Liao ViT 64 8 0 23 May 2024
LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate A. Fuller Daniel G. Kyrollos Yousef Yassin James R. Green 46 2 0 22 May 2024
Vision Transformer with Sparse Scan Prior Qihang Fan Huaibo Huang Mingrui Chen Ran He ViT 48 5 0 22 May 2024
SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalization Jialong Guo Xinghao Chen Yehui Tang Yunhe Wang ViT 49 9 0 19 May 2024
$${M^2D}$NeRF: Multi-Modal Decomposition NeRF with 3D Feature Fields$ ${M^2D}$ NeRF: Multi-Modal Decomposition NeRF with 3D Feature Fields N. Wang Lefei Zhang Angel X Chang 48 0 0 08 May 2024
S3Former: Self-supervised High-resolution Transformer for Solar PV Profiling Minh-Triet Tran Adrian de Luis Haitao Liao Ying Huang Roy McCann Alan Mantooth Jack Cothren Ngan Le 87 0 0 07 May 2024
Key Patches Are All You Need: A Multiple Instance Learning Framework For Robust Medical Diagnosis Diogo J. Araújo M. R. Verdelho Alceu Bissoto Jacinto C. Nascimento Carlos Santiago Catarina Barata 27 1 0 02 May 2024
ShadowMaskFormer: Mask Augmented Patch Embeddings for Shadow Removal Zhuohao Li Guoyang Xie Guannan Jiang Zhichao Lu 33 3 0 29 Apr 2024
Embracing Diversity: Interpretable Zero-shot classification beyond one vector per class Mazda Moayeri Michael G. Rabbat Mark Ibrahim Diane Bouchacourt VLM 46 1 0 25 Apr 2024
Understanding Hyperbolic Metric Learning through Hard Negative Sampling Yun Yue Fangzhou Lin Guanyi Mou Ziming Zhang SSL 30 1 0 23 Apr 2024
StrideNET: Swin Transformer for Terrain Recognition with Dynamic Roughness Extraction Maitreya Shelare Neha Shigvan Atharva Satam Poonam Sonar 14 1 0 20 Apr 2024
AMU-Tuning: Effective Logit Bias for CLIP-based Few-shot Learning Yuwei Tang Zhenyi Lin Qilong Wang Pengfei Zhu Qinghua Hu 30 11 0 13 Apr 2024
DWE+: Dual-Way Matching Enhanced Framework for Multimodal Entity Linking Shezheng Song Shasha Li Shan Zhao Xiaopeng Li Chengyu Wang Jie Yu Jun Ma Tianwei Yan Bing Ji Xiaoguang Mao 23 0 0 07 Apr 2024
A Multi-Level Framework for Accelerating Training Transformer Models Longwei Zou Han Zhang Yangdong Deng AI4CE 34 1 0 07 Apr 2024
Dissecting Query-Key Interaction in Vision Transformers Xu Pan Aaron Philip Ziqian Xie Odelia Schwartz 39 1 0 04 Apr 2024
Instance-Aware Group Quantization for Vision Transformers Jaehyeon Moon Dohyung Kim Junyong Cheon Bumsub Ham MQ ViT 27 6 0 01 Apr 2024
Training-Free Semantic Segmentation via LLM-Supervision Wenfang Sun Yingjun Du Gaowen Liu Ramana Rao Kompella Cees G. M. Snoek VLM 44 2 0 31 Mar 2024
SpiralMLP: A Lightweight Vision MLP Architecture Haojie Mu Burhan Ul Tayyab Nicholas Chua 43 0 0 31 Mar 2024
FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Models Barbara Toniella Corradini Mustafa Shukor Paul Couairon Guillaume Couairon Franco Scarselli Matthieu Cord DiffM VLM 45 4 0 29 Mar 2024
Efficient Modulation for Vision Networks Xu Ma Xiyang Dai Jianwei Yang Bin Xiao Yinpeng Chen Yun Fu Lu Yuan 43 17 0 29 Mar 2024
Enhancing Efficiency in Vision Transformer Networks: Design Techniques and Insights Moein Heidari Reza Azad Sina Ghorbani Kolahi René Arimond Leon Niggemeier ... Afshin Bozorgpour Ehsan Khodapanah Aghdam A. Kazerouni I. Hacihaliloglu Dorit Merhof 45 7 0 28 Mar 2024
Surgical-LVLM: Learning to Adapt Large Vision-Language Model for Grounded Visual Question Answering in Robotic Surgery Guan-Feng Wang Long Bai Wan Jun Nah Jie Wang Zhaoxi Zhang Zhen Chen Jinlin Wu Mobarakol Islam Hongbin Liu Hongliang Ren 46 14 0 22 Mar 2024
Learning to Project for Cross-Task Knowledge Distillation Dylan Auty Roy Miles Benedikt Kolbeinsson K. Mikolajczyk 40 0 0 21 Mar 2024
Boosting Transferability in Vision-Language Attacks via Diversification along the Intersection Region of Adversarial Trajectory Sensen Gao Xiaojun Jia Xuhong Ren Ivor Tsang Qing-Wu Guo AAML 38 14 0 19 Mar 2024
ADAPT to Robustify Prompt Tuning Vision Transformers Masih Eskandar Tooba Imtiaz Zifeng Wang Jennifer Dy VPVLM VLM AAML 36 0 0 19 Mar 2024
HIRI-ViT: Scaling Vision Transformer with High Resolution Inputs Ting Yao Yehao Li Yingwei Pan Tao Mei ViT 25 15 0 18 Mar 2024
A Decade's Battle on Dataset Bias: Are We There Yet? Zhuang Liu Kaiming He 42 28 0 13 Mar 2024
Segmentation Guided Sparse Transformer for Under-Display Camera Image Restoration Jingyun Xue Tao Wang Jun Wang Kaihao Zhang ViT 43 2 0 09 Mar 2024
Tune without Validation: Searching for Learning Rate and Weight Decay on Training Sets Lorenzo Brigato S. Mougiakakou 34 0 0 08 Mar 2024
LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking Jialin Li Qiang Nie Weifu Fu Yuhuan Lin Guangpin Tao Yong-Jin Liu Chengjie Wang 32 4 0 07 Mar 2024
Domain-Agnostic Mutual Prompting for Unsupervised Domain Adaptation Zhekai Du Xinyao Li Fengling Li Ke Lu Lei Zhu Jingjing Li 40 15 0 05 Mar 2024
Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architectures Yuchen Duan Weiyun Wang Zhe Chen Xizhou Zhu Lewei Lu Tong Lu Yu Qiao Hongsheng Li Jifeng Dai Wenhai Wang ViT 46 44 0 04 Mar 2024
LUM-ViT: Learnable Under-sampling Mask Vision Transformer for Bandwidth Limited Optical Signal Acquisition Lingfeng Liu Dong Ni Hangjie Yuan ViT 32 0 0 03 Mar 2024
Large Convolutional Model Tuning via Filter Subspace Wei Chen Zichen Miao Qiang Qiu 51 3 0 01 Mar 2024
Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models Frederik Kunstner Robin Yadav Alan Milligan Mark Schmidt Alberto Bietti 33 26 0 29 Feb 2024
Attention-aware Semantic Communications for Collaborative Inference Jiwoong Im Nayoung Kwon Taewoo Park Jiheon Woo Jaeho Lee Yongjune Kim 46 2 0 23 Feb 2024