Blockwise Self-Attention for Long Document Understanding

7 November 2019

Hao Ma

Sinong Wang

Papers citing "Blockwise Self-Attention for Long Document Understanding"

26 / 76 papers shown

Title
Transformer Acceleration with Dynamic Sparse Attention Liu Liu Zheng Qu Zhaodong Chen Yufei Ding Yuan Xie 24 21 0 21 Oct 2021
Token Pooling in Vision Transformers D. Marin Jen-Hao Rick Chang Anurag Ranjan Anish K. Prabhu Mohammad Rastegari Oncel Tuzel ViT 78 68 0 08 Oct 2021
Greenformers: Improving Computation and Memory Efficiency in Transformer Models via Low-Rank Approximation Samuel Cahyawijaya 34 12 0 24 Aug 2021
FLAT: An Optimized Dataflow for Mitigating Attention Bottlenecks Sheng-Chun Kao Suvinay Subramanian Gaurav Agrawal Amir Yazdanbakhsh T. Krishna 53 59 0 13 Jul 2021
Long-Short Transformer: Efficient Transformers for Language and Vision Chen Zhu Ming-Yu Liu Chaowei Xiao Mohammad Shoeybi Tom Goldstein Anima Anandkumar Bryan Catanzaro ViT VLM 32 132 0 05 Jul 2021
XCiT: Cross-Covariance Image Transformers Alaaeldin El-Nouby Hugo Touvron Mathilde Caron Piotr Bojanowski Matthijs Douze ... Ivan Laptev Natalia Neverova Gabriel Synnaeve Jakob Verbeek Hervé Jégou ViT 47 505 0 17 Jun 2021
Hi-Transformer: Hierarchical Interactive Transformer for Efficient and Effective Long Document Modeling Chuhan Wu Fangzhao Wu Tao Qi Yongfeng Huang 41 67 0 02 Jun 2021
Poolingformer: Long Document Modeling with Pooling Attention Hang Zhang Yeyun Gong Yelong Shen Weisheng Li Jiancheng Lv Nan Duan Weizhu Chen 43 98 0 10 May 2021
T-EMDE: Sketching-based global similarity for cross-modal retrieval Barbara Rychalska Mikolaj Wieczorek Jacek Dąbrowski 38 0 0 10 May 2021
Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents Chaojun Xiao Xueyu Hu Zhiyuan Liu Cunchao Tu Maosong Sun AILaw ELM 68 231 0 09 May 2021
FNet: Mixing Tokens with Fourier Transforms James Lee-Thorp Joshua Ainslie Ilya Eckstein Santiago Ontanon 61 521 0 09 May 2021
Long-Span Summarization via Local Attention and Content Selection Potsawee Manakul Mark Gales 26 42 0 08 May 2021
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding Pengchuan Zhang Xiyang Dai Jianwei Yang Bin Xiao Lu Yuan Lei Zhang Jianfeng Gao ViT 29 331 0 29 Mar 2021
Finetuning Pretrained Transformers into RNNs Jungo Kasai Hao Peng Yizhe Zhang Dani Yogatama Gabriel Ilharco Nikolaos Pappas Yi Mao Weizhu Chen Noah A. Smith 58 63 0 24 Mar 2021
Random Feature Attention Hao Peng Nikolaos Pappas Dani Yogatama Roy Schwartz Noah A. Smith Lingpeng Kong 49 352 0 03 Mar 2021
SparseBERT: Rethinking the Importance Analysis in Self-attention Han Shi Jiahui Gao Xiaozhe Ren Hang Xu Xiaodan Liang Zhenguo Li James T. Kwok 28 54 0 25 Feb 2021
EasyTransfer -- A Simple and Scalable Deep Transfer Learning Platform for NLP Applications Minghui Qiu Peng Li Chengyu Wang Hanjie Pan Yaliang Li ... Jun Yang Yaliang Li Jun Huang Deng Cai Wei Lin VLM SyDa 53 20 0 18 Nov 2020
Zero-shot Entity Linking with Efficient Long Range Sequence Modeling Zonghai Yao Liangliang Cao Huapu Pan VLM 46 21 0 12 Oct 2020
Efficient Transformers: A Survey Yi Tay Mostafa Dehghani Dara Bahri Donald Metzler VLM 123 1,109 0 14 Sep 2020
Compression of Deep Learning Models for Text: A Survey Manish Gupta Puneet Agrawal VLM MedIm AI4CE 29 115 0 12 Aug 2020
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 319 2,041 0 28 Jul 2020
Linformer: Self-Attention with Linear Complexity Sinong Wang Belinda Z. Li Madian Khabsa Han Fang Hao Ma 123 1,668 0 08 Jun 2020
GMAT: Global Memory Augmentation for Transformers Ankit Gupta Jonathan Berant RALM 21 50 0 05 Jun 2020
Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching Liu Yang Mingyang Zhang Cheng Li Michael Bendersky Marc Najork 43 87 0 26 Apr 2020
Longformer: The Long-Document Transformer Iz Beltagy Matthew E. Peters Arman Cohan RALM VLM 41 3,972 0 10 Apr 2020
Sparse Sinkhorn Attention Yi Tay Dara Bahri Liu Yang Donald Metzler Da-Cheng Juan 30 333 0 26 Feb 2020