On the Relationship between Self-Attention and Convolutional Layers

8 November 2019

Papers citing "On the Relationship between Self-Attention and Convolutional Layers"

50 / 269 papers shown

Title
Vision Transformers provably learn spatial structure Samy Jelassi Michael E. Sander Yuan-Fang Li ViT MLT 34 74 0 13 Oct 2022
LieGG: Studying Learned Lie Group Generators A. Moskalev A. Sepliarskaia Ivan Sosnovik A. Smeulders 28 22 0 09 Oct 2022
Towards Flexible Inductive Bias via Progressive Reparameterization Scheduling Yunsung Lee Gyuseong Lee Kwang-seok Ryoo Hyojun Go Jihye Park Seung Wook Kim 32 5 0 04 Oct 2022
Feature Embedding by Template Matching as a ResNet Block Ada Gorgun Y. Z. Gürbüz A. Aydin Alatan 25 1 0 03 Oct 2022
Towards Multi-spatiotemporal-scale Generalized PDE Modeling Jayesh K. Gupta Johannes Brandstetter AI4CE 61 119 0 30 Sep 2022
UNesT: Local Spatial Representation Learning with Hierarchical Transformer for Efficient Medical Segmentation Xin Yu Qi Yang Yinchi Zhou L. Cai Riqiang Gao ... R. Abramson Zizhao Zhang Yuankai Huo Bennett A. Landman Yucheng Tang ViT MedIm 42 0 0 28 Sep 2022
Implementing and Experimenting with Diffusion Models for Text-to-Image Generation Robin Zbinden 30 3 0 22 Sep 2022
gSwin: Gated MLP Vision Model with Hierarchical Structure of Shifted Window Mocho Go Hideyuki Tachibana ViT 37 9 0 24 Aug 2022
MAFNet: A Multi-Attention Fusion Network for RGB-T Crowd Counting Pengyu Chen Junyuan Gao Yuan. Yuan Qi. Wang 22 6 0 14 Aug 2022
Deep is a Luxury We Don't Have Ahmed Taha Yen Nhi Truong Vu Brent Mombourquette Thomas P. Matthews Jason Su Sadanand Singh ViT MedIm 26 2 0 11 Aug 2022
A Length Adaptive Algorithm-Hardware Co-design of Transformer on FPGA Through Sparse Attention and Dynamic Pipelining Hongwu Peng Shaoyi Huang Shiyang Chen Bingbing Li Tong Geng ... Weiwen Jiang Wujie Wen J. Bi Hang Liu Caiwen Ding 47 54 0 07 Aug 2022
Understanding Adversarial Robustness of Vision Transformers via Cauchy Problem Zheng Wang Wenjie Ruan ViT 42 8 0 01 Aug 2022
Convolutional Embedding Makes Hierarchical Vision Transformer Stronger Cong Wang Hongmin Xu Xiong Zhang Li Wang Zhitong Zheng Haifeng Liu ViT 20 20 0 27 Jul 2022
SplitMixer: Fat Trimmed From MLP-like Models Ali Borji Sikun Lin 23 3 0 21 Jul 2022
Single Frame Atmospheric Turbulence Mitigation: A Benchmark Study and A New Physics-Inspired Transformer Model Zhiyuan Mao Ajay Jaiswal Zhangyang Wang Stanley H. Chan 18 51 0 20 Jul 2022
Action Quality Assessment using Transformers Abhay Iyer Mohammad Alali Hemanth Bodala Sunit Vaidya ViT 24 0 0 20 Jul 2022
Parameterization of Cross-Token Relations with Relative Positional Encoding for Vision MLP Zhicai Wang Y. Hao Xingyu Gao Hao Zhang Shuo Wang Tingting Mu Xiangnan He 21 8 0 15 Jul 2022
Vision Transformers: State of the Art and Research Challenges Bo-Kai Ruan Hong-Han Shuai Wen-Huang Cheng ViT 30 17 0 07 Jul 2022
Pure Transformers are Powerful Graph Learners Jinwoo Kim Tien Dat Nguyen Seonwoo Min Sungjun Cho Moontae Lee Honglak Lee Seunghoon Hong 43 189 0 06 Jul 2022
EATFormer: Improving Vision Transformer Inspired by Evolutionary Algorithm Jiangning Zhang Xiangtai Li Yabiao Wang Chengjie Wang Yibo Yang Yong Liu Dacheng Tao ViT 34 32 0 19 Jun 2022
Peripheral Vision Transformer Juhong Min Yucheng Zhao Chong Luo Minsu Cho ViT MDE 32 30 0 14 Jun 2022
Unveiling Transformers with LEGO: a synthetic reasoning task Yi Zhang A. Backurs Sébastien Bubeck Ronen Eldan Suriya Gunasekar Tal Wagner LRM 36 85 0 09 Jun 2022
Blind Face Restoration: Benchmark Datasets and a Baseline Model Puyang Zhang Kaihao Zhang Wenhan Luo Changsheng Li Guoren Wang CVBM 37 17 0 08 Jun 2022
YOLOv5s-GTB: light-weighted and improved YOLOv5s for bridge crack detection Ruiqiang Xiao 11 6 0 03 Jun 2022
Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectives Jun Li Junyu Chen Yucheng Tang Ce Wang Bennett A. Landman S. K. Zhou ViT OOD MedIm 23 21 0 02 Jun 2022
A Survey on Deep Learning for Skin Lesion Segmentation Z. Mirikharaji Kumar Abhishek Alceu Bissoto Catarina Barata Sandra Avila Eduardo Valle M. Celebi Ghassan Hamarneh 39 82 0 01 Jun 2022
MDMLP: Image Classification from Scratch on Small Datasets with MLP Tianxu Lv Chongyang Bai Chaojie Wang 26 5 0 28 May 2022
Your Transformer May Not be as Powerful as You Expect Shengjie Luo Shanda Li Shuxin Zheng Tie-Yan Liu Liwei Wang Di He 70 51 0 26 May 2022
Fast Vision Transformers with HiLo Attention Zizheng Pan Jianfei Cai Bohan Zhuang 28 152 0 26 May 2022
TransforMatcher: Match-to-Match Attention for Semantic Correspondence Seungwook Kim Juhong Min Minsu Cho ViT 46 32 0 23 May 2022
VNT-Net: Rotational Invariant Vector Neuron Transformers Hedi Zisling Andrei Sharf 3DPC 32 1 0 19 May 2022
Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformers Arda Sahiner Tolga Ergen Batu Mehmet Ozturkler John M. Pauly Morteza Mardani Mert Pilanci 40 33 0 17 May 2022
Multi-Head Attention Neural Network for Smartphone Invariant Indoor Localization Saideep Tiku Danish Gufran S. Pasricha 21 15 0 17 May 2022
SeqNet: An Efficient Neural Network for Automatic Malware Detection Jiawei Xu Wenxuan Fu Haoyu Bu Zhi Wang Lingyun Ying AAML 8 3 0 08 May 2022
Attention Mechanism in Neural Networks: Where it Comes and Where it Goes Derya Soydaner 3DV 44 149 0 27 Apr 2022
A survey on attention mechanisms for medical applications: are we moving towards better algorithms? Tiago Gonçalves Isabel Rio-Torto Luís F. Teixeira J. S. Cardoso OOD MedIm 29 36 0 26 Apr 2022
Standing on the Shoulders of Giant Frozen Language Models Yoav Levine Itay Dalmedigos Ori Ram Yoel Zeldes Daniel Jannai ... Barak Lenz Shai Shalev-Shwartz Amnon Shashua Kevin Leyton-Brown Y. Shoham VLM 35 49 0 21 Apr 2022
DeiT III: Revenge of the ViT Hugo Touvron Matthieu Cord Hervé Jégou ViT 48 391 0 14 Apr 2022
VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformers Estelle Aflalo Meng Du Shao-Yen Tseng Yongfei Liu Chenfei Wu Nan Duan Vasudev Lal 33 45 0 30 Mar 2022
Word Order Does Matter (And Shuffled Language Models Know It) Vinit Ravishankar Mostafa Abdou Artur Kulmizev Anders Søgaard 17 44 0 21 Mar 2022
simCrossTrans: A Simple Cross-Modality Transfer Learning for Object Detection with ConvNets or Vision Transformers Xiaoke Shen I. Stamos ViT 18 5 0 20 Mar 2022
CodedVTR: Codebook-based Sparse Voxel Transformer with Geometric Guidance Tianchen Zhao Niansong Zhang Xuefei Ning He Wang Li Yi Yu Wang 3DPC ViT 22 8 0 18 Mar 2022
Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs Xiaohan Ding Xinming Zhang Yi Zhou Jungong Han Guiguang Ding Jian Sun VLM 49 528 0 13 Mar 2022
The Principle of Diversity: Training Stronger Vision Transformers Calls for Reducing All Levels of Redundancy Tianlong Chen Zhenyu (Allen) Zhang Yu Cheng Ahmed Hassan Awadallah Zhangyang Wang ViT 41 37 0 12 Mar 2022
ChiTransformer:Towards Reliable Stereo from Cues Qing Su Shihao Ji MDE ViT 18 12 0 09 Mar 2022
Characterizing Renal Structures with 3D Block Aggregate Transformers Xin Yu Yucheng Tang Yinchi Zhou Riqiang Gao Qi Yang ... Yuankai Huo Zhoubing Xu Thomas A. Lasko R. Abramson Bennett A. Landman MedIm ViT 32 3 0 04 Mar 2022
ViT-P: Rethinking Data-efficient Vision Transformers from Locality Bin Chen Ran A. Wang Di Ming Xin Feng ViT 18 7 0 04 Mar 2022
Aggregated Pyramid Vision Transformer: Split-transform-merge Strategy for Image Recognition without Convolutions Ruikang Ju Ting-Yu Lin Jen-Shiun Chiang Jia-Hao Jian Yu-Shian Lin Liu-Rui-Yi Huang ViT 16 1 0 02 Mar 2022
How Do Vision Transformers Work? Namuk Park Songkuk Kim ViT 47 466 0 14 Feb 2022
Query Efficient Decision Based Sparse Attacks Against Black-Box Deep Learning Models Viet Vo Ehsan Abbasnejad Damith C. Ranasinghe AAML 30 14 0 31 Jan 2022