Simple Local Attentions Remain Competitive for Long-Context Tasks

Simple Local Attentions Remain Competitive for Long-Context Tasks

14 December 2021

Diana Liskovich

Yashar Mehdad

Papers citing "Simple Local Attentions Remain Competitive for Long-Context Tasks"

16 / 16 papers shown

Title
Context Parallelism for Scalable Million-Token Inference Amy Yang Jingyi Yang Aya Ibrahim Xinfeng Xie Bangsheng Tang Grigory Sizov Jeremy Reizenstein Jongsoo Park Jianyu Huang MoE LRM 67 5 0 04 Nov 2024
Training-free Diffusion Model Adaptation for Variable-Sized Text-to-Image Synthesis Zhiyu Jin Xuli Shen Bin Li Xiangyang Xue 24 36 0 14 Jun 2023
Focus Your Attention (with Adaptive IIR Filters) Shahar Lutati Itamar Zimerman Lior Wolf 32 9 0 24 May 2023
DAPR: A Benchmark on Document-Aware Passage Retrieval Kexin Wang Nils Reimers Iryna Gurevych 18 5 0 23 May 2023
Efficient Attention via Control Variates Lin Zheng Jianbo Yuan Chong-Jun Wang Lingpeng Kong 34 18 0 09 Feb 2023
A Length-Extrapolatable Transformer Yutao Sun Li Dong Barun Patra Shuming Ma Shaohan Huang Alon Benhaim Vishrav Chaudhary Xia Song Furu Wei 30 115 0 20 Dec 2022
How Far are We from Robust Long Abstractive Summarization? Huan Yee Koh Jiaxin Ju He Zhang Ming Liu Shirui Pan HILM 28 39 0 30 Oct 2022
CAB: Comprehensive Attention Benchmarking on Long Sequence Modeling Jinchao Zhang Shuyang Jiang Jiangtao Feng Lin Zheng Lingpeng Kong 3DV 43 9 0 14 Oct 2022
An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classification Ilias Chalkidis Xiang Dai Manos Fergadiotis Prodromos Malakasiotis Desmond Elliott 34 33 0 11 Oct 2022
Adapting Pretrained Text-to-Text Models for Long Text Sequences Wenhan Xiong Anchit Gupta Shubham Toshniwal Yashar Mehdad Wen-tau Yih RALM VLM 59 30 0 21 Sep 2022
Efficient Long-Text Understanding with Short-Text Models Maor Ivgi Uri Shaham Jonathan Berant VLM 27 75 0 01 Aug 2022
Revisiting Transformer-based Models for Long Document Classification Xiang Dai Ilias Chalkidis S. Darkner Desmond Elliott VLM 18 68 0 14 Apr 2022
Shortformer: Better Language Modeling using Shorter Inputs Ofir Press Noah A. Smith M. Lewis 230 89 0 31 Dec 2020
Efficient Transformers: A Survey Yi Tay Mostafa Dehghani Dara Bahri Donald Metzler VLM 109 1,102 0 14 Sep 2020
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 285 2,017 0 28 Jul 2020
Efficient Content-Based Sparse Attention with Routing Transformers Aurko Roy M. Saffar Ashish Vaswani David Grangier MoE 252 580 0 12 Mar 2020