Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned

23 May 2019

Papers citing "Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned"

30 / 230 papers shown

Title
Data Movement Is All You Need: A Case Study on Optimizing Transformers A. Ivanov Nikoli Dryden Tal Ben-Nun Shigang Li Torsten Hoefler 36 131 0 30 Jun 2020
Multi-Head Attention: Collaborate Instead of Concatenate Jean-Baptiste Cordonnier Andreas Loukas Martin Jaggi 6 108 0 29 Jun 2020
BERTology Meets Biology: Interpreting Attention in Protein Language Models Jesse Vig Ali Madani Lav Varshney Caiming Xiong R. Socher Nazneen Rajani 29 289 0 26 Jun 2020
On the Computational Power of Transformers and its Implications in Sequence Modeling S. Bhattamishra Arkil Patel Navin Goyal 33 65 0 16 Jun 2020
Roses Are Red, Violets Are Blue... but Should Vqa Expect Them To? Corentin Kervadec G. Antipov M. Baccouche Christian Wolf OOD 21 88 0 09 Jun 2020
Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models Jize Cao Zhe Gan Yu Cheng Licheng Yu Yen-Chun Chen Jingjing Liu VLM 22 127 0 15 May 2020
Hard-Coded Gaussian Attention for Neural Machine Translation Weiqiu You Simeng Sun Mohit Iyyer 33 67 0 02 May 2020
When BERT Plays the Lottery, All Tickets Are Winning Sai Prasanna Anna Rogers Anna Rumshisky MILM 16 186 0 01 May 2020
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference Ji Xin Raphael Tang Jaejun Lee Yaoliang Yu Jimmy J. Lin 17 365 0 27 Apr 2020
The Right Tool for the Job: Matching Model and Instance Complexities Roy Schwartz Gabriel Stanovsky Swabha Swayamdipta Jesse Dodge Noah A. Smith 41 168 0 16 Apr 2020
Information-Theoretic Probing with Minimum Description Length Elena Voita Ivan Titov 23 271 0 27 Mar 2020
Pre-trained Models for Natural Language Processing: A Survey Xipeng Qiu Tianxiang Sun Yige Xu Yunfan Shao Ning Dai Xuanjing Huang LM&MA VLM 246 1,454 0 18 Mar 2020
Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation Alessandro Raganato Yves Scherrer Jörg Tiedemann 32 92 0 24 Feb 2020
Low-Rank Bottleneck in Multi-head Attention Models Srinadh Bhojanapalli Chulhee Yun A. S. Rawat Sashank J. Reddi Sanjiv Kumar 24 94 0 17 Feb 2020
Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction Taeuk Kim Jihun Choi Daniel Edmiston Sang-goo Lee 22 90 0 30 Jan 2020
SANST: A Self-Attentive Network for Next Point-of-Interest Recommendation Qi Guo Jianzhong Qi AI4TS 21 8 0 22 Jan 2020
Cross-Lingual Ability of Multilingual BERT: An Empirical Study Karthikeyan K Zihan Wang Stephen D. Mayhew Dan Roth LRM 36 333 0 17 Dec 2019
Graph Transformer for Graph-to-Sequence Learning Deng Cai W. Lam 32 221 0 18 Nov 2019
What do you mean, BERT? Assessing BERT as a Distributional Semantics Model Timothee Mickus Denis Paperno Mathieu Constant Kees van Deemter 29 45 0 13 Nov 2019
Blockwise Self-Attention for Long Document Understanding J. Qiu Hao Ma Omer Levy Scott Yih Sinong Wang Jie Tang 11 252 0 07 Nov 2019
Reducing Transformer Depth on Demand with Structured Dropout Angela Fan Edouard Grave Armand Joulin 43 584 0 25 Sep 2019
SANVis: Visual Analytics for Understanding Self-Attention Networks Cheonbok Park Inyoup Na Yongjang Jo Sungbok Shin J. Yoo Bum Chul Kwon Jian Zhao Hyungjong Noh Yeonsoo Lee Jaegul Choo HAI 35 38 0 13 Sep 2019
Multi-Granularity Self-Attention for Neural Machine Translation Jie Hao Xing Wang Shuming Shi Jinfeng Zhang Zhaopeng Tu MILM 25 48 0 05 Sep 2019
The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives Elena Voita Rico Sennrich Ivan Titov 215 182 0 03 Sep 2019
Improving Multi-Head Attention with Capsule Networks Shuhao Gu Yang Feng 22 12 0 31 Aug 2019
On Identifiability in Transformers Gino Brunner Yang Liu Damian Pascual Oliver Richter Massimiliano Ciaramita Roger Wattenhofer ViT 30 186 0 12 Aug 2019
VisualBERT: A Simple and Performant Baseline for Vision and Language Liunian Harold Li Mark Yatskar Da Yin Cho-Jui Hsieh Kai-Wei Chang VLM 82 1,919 0 09 Aug 2019
Empirical Evaluation of Sequence-to-Sequence Models for Word Discovery in Low-resource Settings Marcely Zanon Boito Aline Villavicencio Laurent Besacier 17 8 0 29 Jun 2019
What Does BERT Look At? An Analysis of BERT's Attention Kevin Clark Urvashi Khandelwal Omer Levy Christopher D. Manning MILM 120 1,584 0 11 Jun 2019
Analyzing the Structure of Attention in a Transformer Language Model Jesse Vig Yonatan Belinkov 30 357 0 07 Jun 2019