Synthesizer: Rethinking Self-Attention in Transformer Models

2 May 2020

Papers citing "Synthesizer: Rethinking Self-Attention in Transformer Models"

32 / 82 papers shown

Title
Couplformer:Rethinking Vision Transformer with Coupling Attention Map Hai Lan Xihao Wang Xian Wei ViT 28 3 0 10 Dec 2021
Spatio-temporal Relation Modeling for Few-shot Action Recognition Anirudh Thatipelli Sanath Narayan Salman Khan Rao Muhammad Anwer F. Khan Guohao Li ViT 27 88 0 09 Dec 2021
3D Medical Point Transformer: Introducing Convolution to Attention Networks for Medical Point Cloud Analysis Jianhui Yu Chaoyi Zhang Heng Wang Dingxin Zhang Yang Song Tiange Xiang Dongnan Liu Weidong (Tom) Cai ViT MedIm 21 32 0 09 Dec 2021
PointMixer: MLP-Mixer for Point Cloud Understanding Jaesung Choe Chunghyun Park François Rameau Jaesik Park In So Kweon 3DPC 39 98 0 22 Nov 2021
The Efficiency Misnomer Daoyuan Chen Liuyi Yao Dawei Gao Ashish Vaswani Yaliang Li 34 98 0 25 Oct 2021
Token Pooling in Vision Transformers D. Marin Jen-Hao Rick Chang Anurag Ranjan Anish K. Prabhu Mohammad Rastegari Oncel Tuzel ViT 76 66 0 08 Oct 2021
Do Long-Range Language Models Actually Use Long-Range Context? Simeng Sun Kalpesh Krishna Andrew Mattarella-Micke Mohit Iyyer RALM 25 80 0 19 Sep 2021
Primer: Searching for Efficient Transformers for Language Modeling David R. So Wojciech Mañke Hanxiao Liu Zihang Dai Noam M. Shazeer Quoc V. Le VLM 85 152 0 17 Sep 2021
Fastformer: Additive Attention Can Be All You Need Chuhan Wu Fangzhao Wu Tao Qi Yongfeng Huang Xing Xie 46 117 0 20 Aug 2021
Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer Yifan Xu Zhijie Zhang Mengdan Zhang Kekai Sheng Ke Li Weiming Dong Liqing Zhang Changsheng Xu Xing Sun ViT 32 201 0 03 Aug 2021
AutoBERT-Zero: Evolving BERT Backbone from Scratch Jiahui Gao Hang Xu Han Shi Xiaozhe Ren Philip L. H. Yu Xiaodan Liang Xin Jiang Zhenguo Li 21 37 0 15 Jul 2021
The Benchmark Lottery Mostafa Dehghani Yi Tay A. Gritsenko Zhe Zhao N. Houlsby Fernando Diaz Donald Metzler Oriol Vinyals 42 89 0 14 Jul 2021
FLAT: An Optimized Dataflow for Mitigating Attention Bottlenecks Sheng-Chun Kao Suvinay Subramanian Gaurav Agrawal Amir Yazdanbakhsh T. Krishna 38 57 0 13 Jul 2021
A Survey of Transformers Tianyang Lin Yuxin Wang Xiangyang Liu Xipeng Qiu ViT 32 1,087 0 08 Jun 2021
Pay Attention to MLPs Hanxiao Liu Zihang Dai David R. So Quoc V. Le AI4CE 45 651 0 17 May 2021
Poolingformer: Long Document Modeling with Pooling Attention Hang Zhang Yeyun Gong Yelong Shen Weisheng Li Jiancheng Lv Nan Duan Weizhu Chen 35 98 0 10 May 2021
T-EMDE: Sketching-based global similarity for cross-modal retrieval Barbara Rychalska Mikolaj Wieczorek Jacek Dąbrowski 27 0 0 10 May 2021
ResMLP: Feedforward networks for image classification with data-efficient training Hugo Touvron Piotr Bojanowski Mathilde Caron Matthieu Cord Alaaeldin El-Nouby ... Gautier Izacard Armand Joulin Gabriel Synnaeve Jakob Verbeek Hervé Jégou VLM 21 655 0 07 May 2021
MLP-Mixer: An all-MLP Architecture for Vision Ilya O. Tolstikhin N. Houlsby Alexander Kolesnikov Lucas Beyer Xiaohua Zhai ... Andreas Steiner Daniel Keysers Jakob Uszkoreit Mario Lucic Alexey Dosovitskiy 271 2,603 0 04 May 2021
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding Pengchuan Zhang Xiyang Dai Jianwei Yang Bin Xiao Lu Yuan Lei Zhang Jianfeng Gao ViT 27 329 0 29 Mar 2021
Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth Yihe Dong Jean-Baptiste Cordonnier Andreas Loukas 34 373 0 05 Mar 2021
Perceiver: General Perception with Iterative Attention Andrew Jaegle Felix Gimeno Andrew Brock Andrew Zisserman Oriol Vinyals João Carreira VLM ViT MDE 65 973 0 04 Mar 2021
Transformers in Vision: A Survey Salman Khan Muzammal Naseer Munawar Hayat Syed Waqas Zamir F. Khan M. Shah ViT 227 2,430 0 04 Jan 2021
Reservoir Transformers Sheng Shen Alexei Baevski Ari S. Morcos Kurt Keutzer Michael Auli Douwe Kiela 35 17 0 30 Dec 2020
Transformer-based End-to-End Speech Recognition with Local Dense Synthesizer Attention Menglong Xu Shengqiang Li Xiao-Lei Zhang 24 31 0 23 Oct 2020
Cluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency Encoding Shuohang Wang Luowei Zhou Zhe Gan Yen-Chun Chen Yuwei Fang S. Sun Yu Cheng Jingjing Liu 35 28 0 13 Sep 2020
Hopfield Networks is All You Need Hubert Ramsauer Bernhard Schafl Johannes Lehner Philipp Seidl Michael Widrich ... David P. Kreil Michael K Kopp G. Klambauer Johannes Brandstetter Sepp Hochreiter 24 412 0 16 Jul 2020
Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures Julien Launay Iacopo Poli Franccois Boniface Florent Krzakala 30 62 0 23 Jun 2020
Input-independent Attention Weights Are Expressive Enough: A Study of Attention in Self-supervised Audio Transformers Tsung-Han Wu Chun-Chen Hsieh Yen-Hao Chen Po-Han Chi Hung-yi Lee 23 1 0 09 Jun 2020
Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation Alessandro Raganato Yves Scherrer Jörg Tiedemann 24 92 0 24 Feb 2020
A Decomposable Attention Model for Natural Language Inference Ankur P. Parikh Oscar Täckström Dipanjan Das Jakob Uszkoreit 201 1,367 0 06 Jun 2016
Effective Approaches to Attention-based Neural Machine Translation Thang Luong Hieu H. Pham Christopher D. Manning 218 7,926 0 17 Aug 2015