Evolving Attention with Residual Convolutions

20 February 2021

Yujing Wang

Jing Bai

Gao Huang

Papers citing "Evolving Attention with Residual Convolutions"

27 / 27 papers shown

Title
RealFormer: Transformer Likes Residual Attention Ruining He Anirudh Ravula Bhargav Kanagal Joshua Ainslie 36 108 0 21 Dec 2020
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai ... Matthias Minderer G. Heigold Sylvain Gelly Jakob Uszkoreit N. Houlsby ViT 307 40,217 0 22 Oct 2020
Synthesizer: Rethinking Self-Attention in Transformer Models Yi Tay Dara Bahri Donald Metzler Da-Cheng Juan Zhe Zhao Che Zheng 39 334 0 02 May 2020
Lite Transformer with Long-Short Range Attention Zhanghao Wu Zhijian Liu Ji Lin Chengyue Wu Song Han 49 321 0 24 Apr 2020
Reformer: The Efficient Transformer Nikita Kitaev Lukasz Kaiser Anselm Levskaya VLM 143 2,279 0 13 Jan 2020
ERASER: A Benchmark to Evaluate Rationalized NLP Models Jay DeYoung Sarthak Jain Nazneen Rajani Eric P. Lehman Caiming Xiong R. Socher Byron C. Wallace 80 632 0 08 Nov 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 254 19,824 0 23 Oct 2019
Attention is not not Explanation Sarah Wiegreffe Yuval Pinter XAI AAML FAtt 40 901 0 13 Aug 2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy M. Lewis Luke Zettlemoyer Veselin Stoyanov AIMat 387 24,160 0 26 Jul 2019
Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting Shiyang Li Xiaoyong Jin Yao Xuan Xiyou Zhou Wenhu Chen Yu Wang Xifeng Yan AI4TS 51 1,398 0 29 Jun 2019
Stand-Alone Self-Attention in Vision Models Prajit Ramachandran Niki Parmar Ashish Vaswani Irwan Bello Anselm Levskaya Jonathon Shlens VLM SLR ViT 57 1,208 0 13 Jun 2019
Adaptive Attention Span in Transformers Sainbayar Sukhbaatar Edouard Grave Piotr Bojanowski Armand Joulin 52 285 0 19 May 2019
Attention Augmented Convolutional Networks Irwan Bello Barret Zoph Ashish Vaswani Jonathon Shlens Quoc V. Le 121 1,008 0 22 Apr 2019
Attention is not Explanation Sarthak Jain Byron C. Wallace FAtt 87 1,307 0 26 Feb 2019
The Evolved Transformer David R. So Chen Liang Quoc V. Le ViT 74 461 0 30 Jan 2019
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 870 93,936 0 11 Oct 2018
Why Self-Attention? A Targeted Evaluation of Neural Machine Translation Architectures Gongbo Tang Mathias Müller Annette Rios Gonzales Rico Sennrich 32 263 0 27 Aug 2018
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 574 7,080 0 20 Apr 2018
Self-Attention with Relative Position Representations Peter Shaw Jakob Uszkoreit Ashish Vaswani 108 2,269 0 06 Mar 2018
Image Transformer Niki Parmar Ashish Vaswani Jakob Uszkoreit Lukasz Kaiser Noam M. Shazeer Alexander Ku Dustin Tran ViT 82 1,671 0 15 Feb 2018
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 422 129,831 0 12 Jun 2017
Convolutional Sequence to Sequence Learning Jonas Gehring Michael Auli David Grangier Denis Yarats Yann N. Dauphin AIMat 120 3,279 0 08 May 2017
Densely Connected Convolutional Networks Gao Huang Zhuang Liu Laurens van der Maaten Kilian Q. Weinberger PINN 3DV 607 36,599 0 25 Aug 2016
"Why Should I Trust You?": Explaining the Predictions of Any Classifier Marco Tulio Ribeiro Sameer Singh Carlos Guestrin FAtt FaML 478 16,765 0 16 Feb 2016
Deep Residual Learning for Image Recognition Kaiming He Xinming Zhang Shaoqing Ren Jian Sun MedIm 1.3K 192,638 0 10 Dec 2015
Rethinking the Inception Architecture for Computer Vision Christian Szegedy Vincent Vanhoucke Sergey Ioffe Jonathon Shlens Z. Wojna 3DV BDL 416 27,231 0 02 Dec 2015
Adam: A Method for Stochastic Optimization Diederik P. Kingma Jimmy Ba ODL 678 149,474 0 22 Dec 2014