What Dense Graph Do You Need for Self-Attention?

What Dense Graph Do You Need for Self-Attention?

27 May 2022

Qipeng Guo

Xuanjing Huang

Xipeng Qiu

Papers citing "What Dense Graph Do You Need for Self-Attention?"

12 / 12 papers shown

Title
LongT5: Efficient Text-To-Text Transformer for Long Sequences Mandy Guo Joshua Ainslie David C. Uthus Santiago Ontanon Jianmo Ni Yun-hsuan Sung Yinfei Yang VLM 55 312 0 15 Dec 2021
How to Train BERT with an Academic Budget Peter Izsak Moshe Berchansky Omer Levy 82 117 0 15 Apr 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 425 2,081 0 31 Dec 2020
Long Range Arena: A Benchmark for Efficient Transformers Yi Tay Mostafa Dehghani Samira Abnar Songlin Yang Dara Bahri Philip Pham J. Rao Liu Yang Sebastian Ruder Donald Metzler 128 717 0 08 Nov 2020
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 499 2,074 0 28 Jul 2020
Sparse GPU Kernels for Deep Learning Trevor Gale Matei A. Zaharia C. Young Erich Elsen 57 232 0 18 Jun 2020
Longformer: The Long-Document Transformer Iz Beltagy Matthew E. Peters Arman Cohan RALM VLM 124 4,048 0 10 Apr 2020
Axial Attention in Multidimensional Transformers Jonathan Ho Nal Kalchbrenner Dirk Weissenborn Tim Salimans 96 528 0 20 Dec 2019
What Does BERT Look At? An Analysis of BERT's Attention Kevin Clark Urvashi Khandelwal Omer Levy Christopher D. Manning MILM 205 1,592 0 11 Jun 2019
Generating Long Sequences with Sparse Transformers R. Child Scott Gray Alec Radford Ilya Sutskever 90 1,894 0 23 Apr 2019
Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing Auxiliary Sentence Chi Sun Luyao Huang Xipeng Qiu 61 610 0 22 Mar 2019
Star-Transformer Qipeng Guo Xipeng Qiu Pengfei Liu Yunfan Shao Xiangyang Xue Zheng Zhang 69 264 0 25 Feb 2019