Multi-Head Attention: Collaborate Instead of Concatenate

29 June 2020

Papers citing "Multi-Head Attention: Collaborate Instead of Concatenate"

22 / 22 papers shown

Title
MAVEN: Multi-modal Attention for Valence-Arousal Emotion Network Vrushank Ahire Kunal Shah Mudasir Nazir Khan Nikhil Pakhale L. Sookha M. A. Ganaie Abhinav Dhall 114 0 0 16 Mar 2025
Training data-efficient image transformers & distillation through attention Hugo Touvron Matthieu Cord Matthijs Douze Francisco Massa Alexandre Sablayrolles Hervé Jégou ViT 384 6,768 0 23 Dec 2020
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai ... Matthias Minderer G. Heigold Sylvain Gelly Jakob Uszkoreit N. Houlsby ViT 648 41,103 0 22 Oct 2020
Synthesizer: Rethinking Self-Attention in Transformer Models Yi Tay Dara Bahri Donald Metzler Da-Cheng Juan Zhe Zhao Che Zheng 58 336 0 02 May 2020
Talking-Heads Attention Noam M. Shazeer Zhenzhong Lan Youlong Cheng Nan Ding L. Hou 117 80 0 05 Mar 2020
Low-Rank Bottleneck in Multi-head Attention Models Srinadh Bhojanapalli Chulhee Yun A. S. Rawat Sashank J. Reddi Sanjiv Kumar 61 95 0 17 Feb 2020
On the Relationship between Self-Attention and Convolutional Layers Jean-Baptiste Cordonnier Andreas Loukas Martin Jaggi 112 534 0 08 Nov 2019
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter Victor Sanh Lysandre Debut Julien Chaumond Thomas Wolf 232 7,504 0 02 Oct 2019
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations Zhenzhong Lan Mingda Chen Sebastian Goodman Kevin Gimpel Piyush Sharma Radu Soricut SSL AIMat 363 6,455 0 26 Sep 2019
Demystifying the MLPerf Benchmark Suite Snehil Verma Qinzhe Wu Bagus Hanindhito Gunjan Jha E. John R. Radhakrishnan L. John VLM 44 8 0 24 Aug 2019
Stand-Alone Self-Attention in Vision Models Prajit Ramachandran Niki Parmar Ashish Vaswani Irwan Bello Anselm Levskaya Jonathon Shlens VLM SLR ViT 92 1,214 0 13 Jun 2019
Are Sixteen Heads Really Better than One? Paul Michel Omer Levy Graham Neubig MoE 100 1,061 0 25 May 2019
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned Elena Voita David Talbot F. Moiseev Rico Sennrich Ivan Titov 111 1,141 0 23 May 2019
Attention Augmented Convolutional Networks Irwan Bello Barret Zoph Ashish Vaswani Jonathon Shlens Quoc V. Le 132 1,014 0 22 Apr 2019
fairseq: A Fast, Extensible Toolkit for Sequence Modeling Myle Ott Sergey Edunov Alexei Baevski Angela Fan Sam Gross Nathan Ng David Grangier Michael Auli VLM FaML 107 3,149 0 01 Apr 2019
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 1.8K 94,891 0 11 Oct 2018
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 1.1K 7,159 0 20 Apr 2018
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 701 131,652 0 12 Jun 2017
TensorLy: Tensor Learning in Python Jean Kossaifi Yannis Panagakis Anima Anandkumar Maja Pantic 54 355 0 29 Oct 2016
Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications Yong-Deok Kim Eunhyeok Park S. Yoo Taelim Choi Lu Yang Dongjun Shin 124 894 0 20 Nov 2015
ImageNet Large Scale Visual Recognition Challenge Olga Russakovsky Jia Deng Hao Su J. Krause S. Satheesh ... A. Karpathy A. Khosla Michael S. Bernstein Alexander C. Berg Li Fei-Fei VLM ObjD 1.7K 39,525 0 01 Sep 2014
Neural Machine Translation by Jointly Learning to Align and Translate Dzmitry Bahdanau Kyunghyun Cho Yoshua Bengio AIMat 552 27,300 0 01 Sep 2014