Low-Rank Bottleneck in Multi-head Attention Models

17 February 2020

Srinadh Bhojanapalli

Sanjiv Kumar

Papers citing "Low-Rank Bottleneck in Multi-head Attention Models"

32 / 32 papers shown

Title
Approximation Rate of the Transformer Architecture for Sequence Modeling Hao Jiang Qianxiao Li 66 10 0 03 Jan 2025
Hadamard Attention Recurrent Transformer: A Strong Baseline for Stereo Matching Transformer Ziyang Chen Yongjun Zhang Wenting Li Bingshu Wang Yabo Wu Yong Zhao C. L. P. Chen 141 0 0 02 Jan 2025
Data-free Weight Compress and Denoise for Large Language Models Runyu Peng Yunhua Zhou Qipeng Guo Yang Gao Hang Yan Xipeng Qiu Dahua Lin 91 1 0 26 Feb 2024
ViTCoD: Vision Transformer Acceleration via Dedicated Algorithm and Accelerator Co-Design Haoran You Zhanyi Sun Huihong Shi Zhongzhi Yu Yang Zhao Yongan Zhang Chaojian Li Baopu Li Yingyan Lin ViT 56 82 0 18 Oct 2022
Are Transformers universal approximators of sequence-to-sequence functions? Chulhee Yun Srinadh Bhojanapalli A. S. Rawat Sashank J. Reddi Sanjiv Kumar 80 347 0 20 Dec 2019
Adaptively Sparse Transformers Gonçalo M. Correia Vlad Niculae André F. T. Martins 59 253 0 30 Aug 2019
On Identifiability in Transformers Gino Brunner Yang Liu Damian Pascual Oliver Richter Massimiliano Ciaramita Roger Wattenhofer ViT 54 188 0 12 Aug 2019
XLNet: Generalized Autoregressive Pretraining for Language Understanding Zhilin Yang Zihang Dai Yiming Yang J. Carbonell Ruslan Salakhutdinov Quoc V. Le AI4CE 183 8,386 0 19 Jun 2019
Are Sixteen Heads Really Better than One? Paul Michel Omer Levy Graham Neubig MoE 73 1,051 0 25 May 2019
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned Elena Voita David Talbot F. Moiseev Rico Sennrich Ivan Titov 76 1,120 0 23 May 2019
Generating Long Sequences with Sparse Transformers R. Child Scott Gray Alec Radford Ilya Sutskever 73 1,880 0 23 Apr 2019
Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion Hao Sun Xu Tan Jun-Wei Gan Hongzhi Liu Sheng Zhao Tao Qin Tie-Yan Liu 24 65 0 06 Apr 2019
Pay Less Attention with Lightweight and Dynamic Convolutions Felix Wu Angela Fan Alexei Baevski Yann N. Dauphin Michael Auli 56 606 0 29 Jan 2019
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 951 93,936 0 11 Oct 2018
Relational Deep Reinforcement Learning V. Zambaldi David Raposo Adam Santoro V. Bapst Yujia Li ... Victoria Langston Razvan Pascanu M. Botvinick Oriol Vinyals Peter W. Battaglia OffRL 97 219 0 05 Jun 2018
Scaling Neural Machine Translation Myle Ott Sergey Edunov David Grangier Michael Auli AIMat 151 612 0 01 Jun 2018
Self-Attention Generative Adversarial Networks Han Zhang Ian Goodfellow Dimitris N. Metaxas Augustus Odena GAN 111 3,710 0 21 May 2018
Tensor2Tensor for Neural Machine Translation Ashish Vaswani Samy Bengio E. Brevdo François Chollet Aidan Gomez ... Nal Kalchbrenner Niki Parmar Ryan Sepassi Noam M. Shazeer Jakob Uszkoreit 81 528 0 16 Mar 2018
Generating Wikipedia by Summarizing Long Sequences Peter J. Liu Mohammad Saleh Etienne Pot Ben Goodrich Ryan Sepassi Lukasz Kaiser Noam M. Shazeer CVBM 123 786 0 30 Jan 2018
State-of-the-art Speech Recognition With Sequence-to-Sequence Models Chung-Cheng Chiu Tara N. Sainath Yonghui Wu Rohit Prabhavalkar Patrick Nguyen ... Katya Gonina Navdeep Jaitly Yue Liu J. Chorowski M. Bacchiani AI4TS 76 1,150 0 05 Dec 2017
Non-local Neural Networks Xinyu Wang Ross B. Girshick Abhinav Gupta Kaiming He OffRL 213 8,867 0 21 Nov 2017
Breaking the Softmax Bottleneck: A High-Rank RNN Language Model Zhilin Yang Zihang Dai Ruslan Salakhutdinov William W. Cohen BDL 45 367 0 10 Nov 2017
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 443 129,831 0 12 Jun 2017
Convolutional Sequence to Sequence Learning Jonas Gehring Michael Auli David Grangier Denis Yarats Yann N. Dauphin AIMat 122 3,279 0 08 May 2017
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference Adina Williams Nikita Nangia Samuel R. Bowman 397 4,444 0 18 Apr 2017
Deep Reinforcement Learning: An Overview Yuxi Li OffRL VLM 136 1,517 0 25 Jan 2017
Layer Normalization Jimmy Lei Ba J. Kiros Geoffrey E. Hinton 246 10,412 0 21 Jul 2016
SQuAD: 100,000+ Questions for Machine Comprehension of Text Pranav Rajpurkar Jian Zhang Konstantin Lopyrev Percy Liang RALM 151 8,067 0 16 Jun 2016
Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books Yukun Zhu Ryan Kiros R. Zemel Ruslan Salakhutdinov R. Urtasun Antonio Torralba Sanja Fidler 99 2,529 0 22 Jun 2015
Sequence to Sequence Learning with Neural Networks Ilya Sutskever Oriol Vinyals Quoc V. Le AIMat 280 20,491 0 10 Sep 2014
On the Properties of Neural Machine Translation: Encoder-Decoder Approaches Kyunghyun Cho B. V. Merrienboer Dzmitry Bahdanau Yoshua Bengio AI4CE AIMat 157 6,760 0 03 Sep 2014
One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling Ciprian Chelba Tomas Mikolov M. Schuster Qi Ge T. Brants P. Koehn T. Robinson 118 1,099 0 11 Dec 2013