When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations

3 June 2021

Papers citing "When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations"

50 / 91 papers shown

Title
Sharpness-Aware Minimization with Z-Score Gradient Filtering for Neural Networks Juyoung Yun 38 0 0 05 May 2025
Grams: Gradient Descent with Adaptive Momentum Scaling Yang Cao Xiaoyu Li Zhao-quan Song ODL 92 2 0 22 Dec 2024
Meta Curvature-Aware Minimization for Domain Generalization Zhe Chen Yiwen Ye Feilong Tang Yongsheng Pan Yong-quan Xia BDL 197 1 0 16 Dec 2024
Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Training Zhanpeng Zhou Mingze Wang Yuchen Mao Bingrui Li Junchi Yan AAML 62 0 0 14 Oct 2024
Analytic Convolutional Layer: A Step to Analytic Neural Network Jingmao Cui Donglai Tao Linmi Tao Ruiyang Liu Yu Cheng 31 0 0 03 Jul 2024
Improving robustness to corruptions with multiplicative weight perturbations Trung Trinh Markus Heinonen Luigi Acerbi Samuel Kaski 44 0 0 24 Jun 2024
Agnostic Sharpness-Aware Minimization Van-Anh Nguyen Quyen Tran Tuan Truong Thanh-Toan Do Dinh Q. Phung Trung Le 46 0 0 11 Jun 2024
LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate A. Fuller Daniel G. Kyrollos Yousef Yassin James R. Green 52 2 0 22 May 2024
A Hybrid Generative and Discriminative PointNet on Unordered Point Sets Yang Ye Shihao Ji PINN 3DPC 41 0 0 19 Apr 2024
Flatness Improves Backbone Generalisation in Few-shot Classification Rui Li Martin Trapp Marcus Klasson Arno Solin 45 0 0 11 Apr 2024
Dissecting Query-Key Interaction in Vision Transformers Xu Pan Aaron Philip Ziqian Xie Odelia Schwartz 39 1 0 04 Apr 2024
Approximate Nullspace Augmented Finetuning for Robust Vision Transformers Haoyang Liu Aditya Singh Yijiang Li Haohan Wang AAML ViT 36 1 0 15 Mar 2024
Effective Gradient Sample Size via Variation Estimation for Accelerating Sharpness aware Minimization Jiaxin Deng Junbiao Pang Baochang Zhang Tian Wang 45 1 0 24 Feb 2024
Text-to-Code Generation with Modality-relative Pre-training Fenia Christopoulou Guchun Zhang Gerasimos Lampouras AI4TS 18 1 0 08 Feb 2024
Momentum-SAM: Sharpness Aware Minimization without Computational Overhead Marlon Becker Frederick Altrock Benjamin Risse 79 5 0 22 Jan 2024
Critical Influence of Overparameterization on Sharpness-aware Minimization Sungbin Shin Dongyeop Lee Maksym Andriushchenko Namhoon Lee AAML 44 1 0 29 Nov 2023
Learning Human Action Recognition Representations Without Real Humans Howard Zhong Samarth Mishra Donghyun Kim SouYoung Jin Rameswar Panda Hildegard Kuehne Leonid Karlinsky Venkatesh Saligrama Aude Oliva Rogerio Feris 24 3 0 10 Nov 2023
Rethinking Class-incremental Learning in the Era of Large Pre-trained Models via Test-Time Adaptation Imad Eddine Marouf Subhankar Roy Enzo Tartaglione Stéphane Lathuilière CLL 26 3 0 17 Oct 2023
Vision Transformers Need Registers Zilong Chen Maxime Oquab Julien Mairal Huaping Liu ViT 47 312 0 28 Sep 2023
CAIT: Triple-Win Compression towards High Accuracy, Fast Inference, and Favorable Transferability For ViTs Ao Wang Hui Chen Zijia Lin Sicheng Zhao J. Han Guiguang Ding ViT 31 6 0 27 Sep 2023
Why Does Little Robustness Help? Understanding and Improving Adversarial Transferability from Surrogate Training Yechao Zhang Shengshan Hu Leo Yu Zhang Junyu Shi Minghui Li Xiaogeng Liu Wei Wan Hai Jin AAML 22 21 0 15 Jul 2023
A Novel Site-Agnostic Multimodal Deep Learning Model to Identify Pro-Eating Disorder Content on Social Media J. Feldman 27 0 0 06 Jul 2023
ExpPoint-MAE: Better interpretability and performance for self-supervised point cloud transformers Ioannis Romanelis Vlassis Fotis Konstantinos Moustakas Adrian Munteanu ViT 3DPC 21 4 0 19 Jun 2023
Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optima Dongkuk Si Chulhee Yun 28 15 0 16 Jun 2023
DiTTO: A Feature Representation Imitation Approach for Improving Cross-Lingual Transfer Shanu Kumar Abbaraju Soujanya Sandipan Dandapat Sunayana Sitaram Monojit Choudhury VLM 27 1 0 04 Mar 2023
Dropout Reduces Underfitting Zhuang Liu Zhi-Qin John Xu Joseph Jin Zhiqiang Shen Trevor Darrell 37 36 0 02 Mar 2023
mSAM: Micro-Batch-Averaged Sharpness-Aware Minimization Kayhan Behdin Qingquan Song Aman Gupta S. Keerthi Ayan Acharya Borja Ocejo Gregory Dexter Rajiv Khanna D. Durfee Rahul Mazumder AAML 18 7 0 19 Feb 2023
Mixed Hierarchy Network for Image Restoration Huiyu Gao Depeng Dang 29 14 0 19 Feb 2023
Symbolic Discovery of Optimization Algorithms Xiangning Chen Chen Liang Da Huang Esteban Real Kaiyuan Wang ... Xuanyi Dong Thang Luong Cho-Jui Hsieh Yifeng Lu Quoc V. Le 64 352 0 13 Feb 2023
A Survey on Efficient Training of Transformers Bohan Zhuang Jing Liu Zizheng Pan Haoyu He Yuetian Weng Chunhua Shen 31 47 0 02 Feb 2023
Open-Set Likelihood Maximization for Few-Shot Learning Malik Boudiaf Etienne Bennequin Myriam Tami Antoine Toubhans Pablo Piantanida C´eline Hudelot Ismail Ben Ayed BDL 26 10 0 20 Jan 2023
Stability Analysis of Sharpness-Aware Minimization Hoki Kim Jinseong Park Yujin Choi Jaewook Lee 33 12 0 16 Jan 2023
Swin MAE: Masked Autoencoders for Small Datasets Zián Xu Yin Dai Fayu Liu Weibin Chen Yue Liu Li-Li Shi Sheng Liu Yuhang Zhou SyDa MedIm ViT 36 28 0 28 Dec 2022
On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baseline Nicklas Hansen Zhecheng Yuan Yanjie Ze Tongzhou Mu Aravind Rajeswaran H. Su Huazhe Xu Xiaolong Wang 32 65 0 12 Dec 2022
The Birds Need Attention Too: Analysing usage of Self Attention in identifying bird calls in soundscapes Chandra Kanth Nagesh Abhishek Purushothama 24 2 0 14 Nov 2022
Fully-attentive and interpretable: vision and video vision transformers for pain detection Giacomo Fiorentini Itir Onal Ertugrul A. A. Salah MedIm ViT 13 2 0 27 Oct 2022
K-SAM: Sharpness-Aware Minimization at the Speed of SGD Renkun Ni Ping Yeh-Chiang Jonas Geiping Micah Goldblum A. Wilson Tom Goldstein 23 8 0 23 Oct 2022
SQuAT: Sharpness- and Quantization-Aware Training for BERT Zheng Wang Juncheng Billy Li Shuhui Qu Florian Metze Emma Strubell MQ 21 7 0 13 Oct 2022
Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approach Peng Mi Li Shen Tianhe Ren Yiyi Zhou Xiaoshuai Sun Rongrong Ji Dacheng Tao AAML 27 69 0 11 Oct 2022
SynBench: Task-Agnostic Benchmarking of Pretrained Representations using Synthetic Data Ching-Yun Ko Pin-Yu Chen Jeet Mohapatra Payel Das Lucani E. Daniel 21 3 0 06 Oct 2022
Feature Embedding by Template Matching as a ResNet Block Ada Gorgun Y. Z. Gürbüz Aydin Alatan 22 1 0 03 Oct 2022
Transfer Learning with Pretrained Remote Sensing Transformers A. Fuller K. Millard J.R. Green 30 11 0 28 Sep 2022
Greybox XAI: a Neural-Symbolic learning framework to produce interpretable predictions for image classification Adrien Bennetot Gianni Franchi Javier Del Ser Raja Chatila Natalia Díaz Rodríguez AAML 32 29 0 26 Sep 2022
Colonoscopy Landmark Detection using Vision Transformers Aniruddha Tamhane Tseéla Mida Erez Posner Moshe Bouhnik ViT MedIm 56 5 0 22 Sep 2022
Design of the topology for contrastive visual-textual alignment Zhun Sun 30 1 0 05 Sep 2022
ProtoPFormer: Concentrating on Prototypical Parts in Vision Transformers for Interpretable Image Recognition Mengqi Xue Qihan Huang Haofei Zhang Lechao Cheng Jie Song Ming-hui Wu Mingli Song ViT 25 53 0 22 Aug 2022
Multiple Instance Neuroimage Transformer Ayush Singla Qingyu Zhao Daniel K. Do Yuyin Zhou K. Pohl Ehsan Adeli ViT MedIm 21 11 0 19 Aug 2022
Deep is a Luxury We Don't Have Ahmed Taha Yen Nhi Truong Vu Brent Mombourquette Thomas P. Matthews Jason Su Sadanand Singh ViT MedIm 20 2 0 11 Aug 2022
Exploring the Design of Adaptation Protocols for Improved Generalization and Machine Learning Safety Puja Trivedi Danai Koutra Jayaraman J. Thiagarajan AAML 28 0 0 26 Jul 2022
PanGu-Coder: Program Synthesis with Function-Level Language Modeling Fenia Christopoulou Gerasimos Lampouras Milan Gritta Guchun Zhang Yinpeng Guo ... Guangtai Liang Jia Wei Xin Jiang Qianxiang Wang Qun Liu ELM SyDa ALM 45 74 0 22 Jul 2022