On the Relationship between Self-Attention and Convolutional Layers

8 November 2019

Papers citing "On the Relationship between Self-Attention and Convolutional Layers"

50 / 269 papers shown

Title
YOLOPose V2: Understanding and Improving Transformer-based 6D Pose Estimation Arul Selvam Periyasamy Arash A. Amini Vladimir Tsaturyan Sven Behnke ViT 30 16 0 21 Jul 2023
MSViT: Dynamic Mixed-Scale Tokenization for Vision Transformers Jakob Drachmann Havtorn Amelie Royer Tijmen Blankevoort B. Bejnordi 30 8 0 05 Jul 2023
Concurrent ischemic lesion age estimation and segmentation of CT brain using a Transformer-based network A. Marcus P. Bentley Daniel Rueckert MedIm 21 9 0 21 Jun 2023
$E(2)$ -Equivariant Vision Transformer Renjun Xu Kaifan Yang Ke Liu Fengxiang He ViT MDE 21 9 0 11 Jun 2023
Connectional-Style-Guided Contextual Representation Learning for Brain Disease Diagnosis Gongshu Wang Ning Jiang Yunxiao Ma Tiantian Liu Duanduan Chen Jinglong Wu Guoqi Li Dong Liang Tianyi Yan MedIm 28 2 0 08 Jun 2023
Infusing Lattice Symmetry Priors in Attention Mechanisms for Sample-Efficient Abstract Geometric Reasoning Mattia Atzeni Mrinmaya Sachan Andreas Loukas LRM 24 3 0 05 Jun 2023
Understanding MLP-Mixer as a Wide and Sparse MLP Tomohiro Hayase Ryo Karakida MoE 28 4 0 02 Jun 2023
Ghost Noise for Regularizing Deep Neural Networks Atli Kosson Dongyang Fan Martin Jaggi 25 1 0 26 May 2023
Multiplication-Free Transformer Training via Piecewise Affine Operations Atli Kosson Martin Jaggi 21 4 0 26 May 2023
Evolution: A Unified Formula for Feature Operators from a High-level Perspective Zhicheng Cai 16 0 0 23 May 2023
Mimetic Initialization of Self-Attention Layers Asher Trockman J. Zico Kolter 30 30 0 16 May 2023
Understanding Gaussian Attention Bias of Vision Transformers Using Effective Receptive Fields Bum Jun Kim Hyeyeon Choi Hyeonah Jang Sang Woo Kim ViT 26 3 0 08 May 2023
Real-World Denoising via Diffusion Model Chenghan Yang Lijing Liang Zhixun Su DiffM 24 14 0 08 May 2023
AttentionViz: A Global View of Transformer Attention Catherine Yeh Yida Chen Aoyu Wu Cynthia Chen Fernanda Viégas Martin Wattenberg ViT 33 52 0 04 May 2023
OctFormer: Octree-based Transformers for 3D Point Clouds Peng-Shuai Wang ViT 3DPC 32 82 0 04 May 2023
Preserving Locality in Vision Transformers for Class Incremental Learning Bowen Zheng Da-Wei Zhou Han-Jia Ye De-Chuan Zhan CLL 27 5 0 14 Apr 2023
Inductive biases in deep learning models for weather prediction Jannik Thümmel Matthias Karlbauer S. Otte C. Zarfl Georg Martius ... Thomas Scholten Ulrich Friedrich V. Wulfmeyer B. Goswami Martin Volker Butz AI4CE 43 6 0 06 Apr 2023
Transformer-based interpretable multi-modal data fusion for skin lesion classification Theodor Cheslerean-Boghiu Melia-Evelina Fleischmann Theresa Willem Tobias Lasser ViT MedIm AI4CE 24 2 0 03 Apr 2023
Self-positioning Point-based Transformer for Point Cloud Understanding Jinyoung Park S. Lee S. Kim Yunyang Xiong Hyunwoo J. Kim ViT 3DPC 35 57 0 29 Mar 2023
Transformer-based Multi-Instance Learning for Weakly Supervised Object Detection Zhaofei Wang Weijia Zhang Min-Ling Zhang ViT WSOD 21 3 0 27 Mar 2023
Robustifying Token Attention for Vision Transformers Yong Guo David Stutz Bernt Schiele ViT 21 24 0 20 Mar 2023
DS-TDNN: Dual-stream Time-delay Neural Network with Global-aware Filter for Speaker Verification Yangfu Li Jiapan Gan Xiaodan Lin 24 6 0 20 Mar 2023
CerviFormer: A Pap-smear based cervical cancer classification method using cross attention and latent transformer Bhaswati Singha Deo M. Pal P. Panigrahi A. Pradhan MedIm 36 22 0 17 Mar 2023
Self-attention in Vision Transformers Performs Perceptual Grouping, Not Attention Paria Mehrani John K. Tsotsos 25 24 0 02 Mar 2023
A Brief Survey on the Approximation Theory for Sequence Modelling Hao Jiang Qianxiao Li Zhong Li Shida Wang AI4TS 30 12 0 27 Feb 2023
An algorithmic framework for the optimization of deep neural networks architectures and hyperparameters Julie Keisler El-Ghazali Talbi Sandra Claudel Gilles Cabriel 28 6 0 27 Feb 2023
STOA-VLP: Spatial-Temporal Modeling of Object and Action for Video-Language Pre-training Weihong Zhong Mao Zheng Duyu Tang Xuan Luo Heng Gong Xiaocheng Feng Bing Qin 32 8 0 20 Feb 2023
VITAL: Vision Transformer Neural Networks for Accurate Smartphone Heterogeneity Resilient Indoor Localization Danish Gufran Saideep Tiku S. Pasricha 22 11 0 18 Feb 2023
From paintbrush to pixel: A review of deep neural networks in AI-generated art Anne-Sofie Maerten Derya Soydaner 37 23 0 14 Feb 2023
Multi-modal Machine Learning in Engineering Design: A Review and Future Directions Binyang Song Ruilin Zhou Faez Ahmed AI4CE 37 40 0 14 Feb 2023
A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexity Hongkang Li Hao Wu Sijia Liu Pin-Yu Chen ViT MLT 37 57 0 12 Feb 2023
Scaling Vision Transformers to 22 Billion Parameters Mostafa Dehghani Josip Djolonga Basil Mustafa Piotr Padlewski Jonathan Heek ... Mario Luvcić Xiaohua Zhai Daniel Keysers Jeremiah Harmsen N. Houlsby MLLM 63 572 0 10 Feb 2023
Invariant Slot Attention: Object Discovery with Slot-Centric Reference Frames Ondrej Biza Sjoerd van Steenkiste Mehdi S. M. Sajjadi Gamaleldin F. Elsayed Aravindh Mahendran Thomas Kipf OCL 48 32 0 09 Feb 2023
Adapting Pre-trained Vision Transformers from 2D to 3D through Weight Inflation Improves Medical Image Segmentation Yuhui Zhang Shihua Huang Zhengping Zhou M. Lungren Serena Yeung ViT MedIm 16 8 0 08 Feb 2023
Knowledge Distillation in Vision Transformers: A Critical Review Gousia Habib Tausifa Jan Saleem Brejesh Lall 29 15 0 04 Feb 2023
A Survey on Efficient Training of Transformers Bohan Zhuang Jing Liu Zizheng Pan Haoyu He Yuetian Weng Chunhua Shen 31 47 0 02 Feb 2023
Enhancing Face Recognition with Latent Space Data Augmentation and Facial Posture Reconstruction Soroush Hashemifar Abdolreza Marefat Javad Hassannataj Joloudari H. Hassanpour CVBM 31 11 0 27 Jan 2023
Transformers in Action Recognition: A Review on Temporal Modeling Elham Shabaninia Hossein Nezamabadi-pour Fatemeh Shafizadegan ViT 24 8 0 29 Dec 2022
Part-guided Relational Transformers for Fine-grained Visual Recognition Yifan Zhao Jia Li Xiaowu Chen Yonghong Tian ViT 36 34 0 28 Dec 2022
A Close Look at Spatial Modeling: From Attention to Convolution Xu Ma Huan Wang Can Qin Kunpeng Li Xing Zhao Jie Fu Yun Fu ViT 3DPC 25 11 0 23 Dec 2022
From Xception to NEXcepTion: New Design Decisions and Neural Architecture Search Hadar Shavit Filip Jatelnicki Pol Mor-Puigventós W. Kowalczyk 31 2 0 16 Dec 2022
Adaptive Attention Link-based Regularization for Vision Transformers Heegon Jin Jongwon Choi ViT 22 0 0 25 Nov 2022
Complex-Valued Time-Frequency Self-Attention for Speech Dereverberation Vinay Kothapally John H. L. Hansen 31 9 0 22 Nov 2022
Vision Transformers in Medical Imaging: A Review Emerald U. Henry Onyeka Emebob C. Omonhinmin ViT MedIm 34 34 0 18 Nov 2022
Prompt Tuning for Parameter-efficient Medical Image Segmentation Marc Fischer Alexander Bartler Bin Yang SSeg 24 18 0 16 Nov 2022
Demystify Self-Attention in Vision Transformers from a Semantic Perspective: Analysis and Application Leijie Wu Song Guo Yaohong Ding Junxiao Wang Wenchao Xu Richard Yi Da Xu Jiewei Zhang 41 2 0 13 Nov 2022
Attention-based Neural Cellular Automata Mattie Tesfaldet Derek Nowrouzezahrai C. Pal ViT 37 17 0 02 Nov 2022
Multi-Viewpoint and Multi-Evaluation with Felicitous Inductive Bias Boost Machine Abstract Reasoning Ability Qinglai Wei Diancheng Chen Beiming Yuan 32 10 0 26 Oct 2022
Delving into Masked Autoencoders for Multi-Label Thorax Disease Classification Junfei Xiao Yutong Bai Alan Yuille Zongwei Zhou MedIm ViT 37 59 0 23 Oct 2022
Scratching Visual Transformer's Back with Uniform Attention Nam Hyeon-Woo Kim Yu-Ji Byeongho Heo Doonyoon Han Seong Joon Oh Tae-Hyun Oh 364 23 0 16 Oct 2022