On the Relationship between Self-Attention and Convolutional Layers

8 November 2019

Papers citing "On the Relationship between Self-Attention and Convolutional Layers"

50 / 269 papers shown

Title
Extreme precipitation forecasting using attention augmented convolutions Weichen Huang 23 1 0 31 Jan 2022
Research on Patch Attentive Neural Process Xiaohan Yu Shao‐Chen Mao 19 1 0 29 Jan 2022
Explore-And-Match: Bridging Proposal-Based and Proposal-Free With Transformer for Sentence Grounding in Videos Sangmin Woo Jinyoung Park Inyong Koo Sumin Lee Minki Jeong Changick Kim 44 3 0 25 Jan 2022
Patches Are All You Need? Asher Trockman J. Zico Kolter ViT 225 402 0 24 Jan 2022
UniFormer: Unifying Convolution and Self-attention for Visual Recognition Kunchang Li Yali Wang Junhao Zhang Peng Gao Guanglu Song Yu Liu Hongsheng Li Yu Qiao ViT 162 360 0 24 Jan 2022
Representing Long-Range Context for Graph Neural Networks with Global Attention Zhanghao Wu Paras Jain Matthew A. Wright Azalia Mirhoseini Joseph E. Gonzalez Ion Stoica GNN 46 258 0 21 Jan 2022
Real-World Graph Convolution Networks (RW-GCNs) for Action Recognition in Smart Video Surveillance Justin Sanchez Christopher Neff Hamed Tabkhi GNN 30 9 0 15 Jan 2022
UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning Kunchang Li Yali Wang Peng Gao Guanglu Song Yu Liu Hongsheng Li Yu Qiao ViT 47 238 0 12 Jan 2022
Lawin Transformer: Improving Semantic Segmentation Transformer with Multi-Scale Representations via Large Window Attention Haotian Yan Chuang Zhang Ming Wu ViT 30 63 0 05 Jan 2022
APRIL: Finding the Achilles' Heel on Privacy for Vision Transformers Jiahao Lu Xi Sheryl Zhang Tianli Zhao Xiangyu He Jian Cheng ViT PILM 25 22 0 28 Dec 2021
StyleSwin: Transformer-based GAN for High-resolution Image Generation Bo Zhang Shuyang Gu Bo Zhang Jianmin Bao Dong Chen Fang Wen Yong Wang B. Guo ViT 38 223 0 20 Dec 2021
Deep ViT Features as Dense Visual Descriptors Shirzad Amir Yossi Gandelsman Shai Bagon Tali Dekel MDE ViT 36 274 0 10 Dec 2021
Couplformer:Rethinking Vision Transformer with Coupling Attention Map Hai Lan Xihao Wang Xian Wei ViT 34 3 0 10 Dec 2021
Bootstrapping ViTs: Towards Liberating Vision Transformers from Pre-training Haofei Zhang Jiarui Duan Mengqi Xue Mingli Song Li Sun Xiuming Zhang ViT AI4CE 30 16 0 07 Dec 2021
Dynamic Token Normalization Improves Vision Transformers Wenqi Shao Yixiao Ge Zhaoyang Zhang Xuyuan Xu Xiaogang Wang Ying Shan Ping Luo ViT 123 11 0 05 Dec 2021
On the Integration of Self-Attention and Convolution Xuran Pan Chunjiang Ge Rui Lu S. Song Guanfu Chen Zeyi Huang Gao Huang SSL 41 288 0 29 Nov 2021
Multi-domain Integrative Swin Transformer network for Sparse-View Tomographic Reconstruction Jiayi Pan Heye Zhang Weifei Wu Z. Gao Weiwen Wu 20 59 0 28 Nov 2021
Neural Collaborative Graph Machines for Table Structure Recognition Hao Liu Xin Li Bin Liu Deqiang Jiang Yinsong Liu Bo Ren LMTD 24 31 0 26 Nov 2021
Pruning Self-attentions into Convolutional Layers in Single Path Haoyu He Jianfei Cai Jing Liu Zizheng Pan Jing Zhang Dacheng Tao Bohan Zhuang ViT 34 40 0 23 Nov 2021
Mesa: A Memory-saving Training Framework for Transformers Zizheng Pan Peng Chen Haoyu He Jing Liu Jianfei Cai Bohan Zhuang 31 20 0 22 Nov 2021
Full-attention based Neural Architecture Search using Context Auto-regression Yuan Zhou Haiyang Wang Shuwei Huo Boyu Wang 30 3 0 13 Nov 2021
A Survey of Visual Transformers Yang Liu Yao Zhang Yixin Wang Feng Hou Jin Yuan Jiang Tian Yang Zhang Zhongchao Shi Jianping Fan Zhiqiang He 3DGS ViT 77 330 0 11 Nov 2021
Revisiting spatio-temporal layouts for compositional action recognition Gorjan Radevski Marie-Francine Moens Tinne Tuytelaars 32 26 0 02 Nov 2021
Can Vision Transformers Perform Convolution? Shanda Li Xiangning Chen Di He Cho-Jui Hsieh ViT 49 19 0 02 Nov 2021
ASFormer: Transformer for Action Segmentation Fangqiu Yi Hongyu Wen Tingting Jiang ViT 79 172 0 16 Oct 2021
FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizes David W. Romero Robert-Jan Bruintjes Jakub M. Tomczak Erik J. Bekkers Mark Hoogendoorn Jan van Gemert 80 82 0 15 Oct 2021
Dynamic Inference with Neural Interpreters Nasim Rahaman Muhammad Waleed Gondal S. Joshi Peter V. Gehler Yoshua Bengio Francesco Locatello Bernhard Schölkopf 48 31 0 12 Oct 2021
Localizing Objects with Self-Supervised Transformers and no Labels Oriane Siméoni Gilles Puy Huy V. Vo Simon Roburin Spyros Gidaris Andrei Bursuc P. Pérez Renaud Marlet Jean Ponce ViT 180 196 0 29 Sep 2021
Learning the Physics of Particle Transport via Transformers O. Pastor-Serrano Zoltán Perkó MedIm 21 13 0 08 Sep 2021
DeepGene Transformer: Transformer for the gene expression-based classification of cancer subtypes Anwar Khan Boreom Lee MedIm 14 16 0 26 Aug 2021
SwinIR: Image Restoration Using Swin Transformer Christos Sakaridis Jie Cao Guolei Sun Kaicheng Zhang Luc Van Gool Radu Timofte ViT 77 2,818 0 23 Aug 2021
Do Vision Transformers See Like Convolutional Neural Networks? M. Raghu Thomas Unterthiner Simon Kornblith Chiyuan Zhang Alexey Dosovitskiy ViT 67 925 0 19 Aug 2021
Few-shot Segmentation with Optimal Transport Matching and Message Flow Weide Liu Chi Zhang Henghui Ding Tzu-Yi Hung Guosheng Lin 27 49 0 19 Aug 2021
RaftMLP: How Much Can Be Done Without Attention and with Less Spatial Locality? Yuki Tatsunami Masato Taki 36 12 0 09 Aug 2021
Global Self-Attention as a Replacement for Graph Convolution Md Shamim Hussain Mohammed J. Zaki D. Subramanian ViT 40 123 0 07 Aug 2021
Sparse Continuous Distributions and Fenchel-Young Losses André F. T. Martins Marcos Vinícius Treviso António Farinhas P. Aguiar Mário A. T. Figueiredo Mathieu Blondel Vlad Niculae 28 11 0 04 Aug 2021
Log-Polar Space Convolution for Convolutional Neural Networks Bing-Huang Su Ji-Rong Wen 19 1 0 26 Jul 2021
CycleMLP: A MLP-like Architecture for Dense Prediction Shoufa Chen Enze Xie Chongjian Ge Runjian Chen Ding Liang Ping Luo 33 231 0 21 Jul 2021
Transformer with Peak Suppression and Knowledge Guidance for Fine-grained Image Recognition Xinda Liu Lili Wang Xiaoguang Han ViT 41 66 0 14 Jul 2021
Visual Parser: Representing Part-whole Hierarchies with Transformers Shuyang Sun Xiaoyu Yue S. Bai Philip Torr 50 27 0 13 Jul 2021
Can Transformers Jump Around Right in Natural Language? Assessing Performance Transfer from SCAN Rahma Chaabouni Roberto Dessì Eugene Kharitonov 27 20 0 03 Jul 2021
Polarized Self-Attention: Towards High-quality Pixel-wise Regression Huajun Liu Fuqiang Liu Xinyi Fan Dong Huang 79 211 0 02 Jul 2021
Early Convolutions Help Transformers See Better Tete Xiao Mannat Singh Eric Mintun Trevor Darrell Piotr Dollár Ross B. Girshick 20 753 0 28 Jun 2021
Co-advise: Cross Inductive Bias Distillation Sucheng Ren Zhengqi Gao Tianyu Hua Zihui Xue Yonglong Tian Shengfeng He Hang Zhao 49 53 0 23 Jun 2021
TokenLearner: What Can 8 Learned Tokens Do for Images and Videos? Michael S. Ryoo A. Piergiovanni Anurag Arnab Mostafa Dehghani A. Angelova ViT 37 127 0 21 Jun 2021
Visual Correspondence Hallucination Hugo Germain Vincent Lepetit Guillaume Bourmaud 38 11 0 17 Jun 2021
GroupBERT: Enhanced Transformer Architecture with Efficient Grouped Structures Ivan Chelombiev Daniel Justus Douglas Orr A. Dietrich Frithjof Gressmann A. Koliousis Carlo Luschi 27 5 0 10 Jun 2021
Transformed CNNs: recasting pre-trained convolutional layers with self-attention Stéphane dÁscoli Levent Sagun Giulio Biroli Ari S. Morcos ViT 18 6 0 10 Jun 2021
Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Models Tyler A. Chang Yifan Xu Weijian Xu Z. Tu ViT 29 15 0 10 Jun 2021
Scaling Vision Transformers Xiaohua Zhai Alexander Kolesnikov N. Houlsby Lucas Beyer ViT 79 1,060 0 08 Jun 2021