Long Range Arena: A Benchmark for Efficient Transformers

8 November 2020

Papers citing "Long Range Arena: A Benchmark for Efficient Transformers"

50 / 184 papers shown

Title
Lost in the Middle: How Language Models Use Long Contexts Nelson F. Liu Kevin Lin John Hewitt Ashwin Paranjape Michele Bevilacqua Fabio Petroni Percy Liang RALM 40 1,404 0 06 Jul 2023
LongNet: Scaling Transformers to 1,000,000,000 Tokens Jiayu Ding Shuming Ma Li Dong Xingxing Zhang Shaohan Huang Wenhui Wang Nanning Zheng Furu Wei CLL 41 151 0 05 Jul 2023
MNISQ: A Large-Scale Quantum Circuit Dataset for Machine Learning on/for Quantum Computers in the NISQ era Leonardo Placidi Ryuichiro Hataya Toshio Mori Koki Aoyama Hayata Morisaki K. Mitarai Keisuke Fujii 21 8 0 29 Jun 2023
When to Use Efficient Self Attention? Profiling Text, Speech and Image Transformer Variants Anuj Diwan Eunsol Choi David Harwath 41 0 0 14 Jun 2023
Focus Your Attention (with Adaptive IIR Filters) Shahar Lutati Itamar Zimerman Lior Wolf 32 9 0 24 May 2023
Fourier Transformer: Fast Long Range Modeling by Removing Sequence Redundancy with FFT Operator Ziwei He Meng-Da Yang Minwei Feng Jingcheng Yin Xinbing Wang Jingwen Leng Zhouhan Lin ViT 35 13 0 24 May 2023
RWKV: Reinventing RNNs for the Transformer Era Bo Peng Eric Alcaide Quentin G. Anthony Alon Albalak Samuel Arcadinho ... Qihang Zhao P. Zhou Qinghua Zhou Jian Zhu Rui-Jie Zhu 90 562 0 22 May 2023
CageViT: Convolutional Activation Guided Efficient Vision Transformer Hao Zheng Jinbao Wang Xiantong Zhen Hao Chen Jingkuan Song Feng Zheng ViT 20 0 0 17 May 2023
Ray-Patch: An Efficient Querying for Light Field Transformers T. B. Martins Javier Civera ViT 39 0 0 16 May 2023
DLUE: Benchmarking Document Language Understanding Ruoxi Xu Hongyu Lin Xinyan Guan Xianpei Han Yingfei Sun Le Sun ELM 24 0 0 16 May 2023
SKI to go Faster: Accelerating Toeplitz Neural Networks via Asymmetric Kernels Alexander Moreno Jonathan Mei Luke Walters 21 0 0 15 May 2023
Vcc: Scaling Transformers to 128K Tokens or More by Prioritizing Important Tokens Zhanpeng Zeng Cole Hawkins Min-Fong Hong Aston Zhang Nikolaos Pappas Vikas Singh Shuai Zheng 21 6 0 07 May 2023
The Role of Global and Local Context in Named Entity Recognition Arthur Amalvy Vincent Labatut Richard Dufour 38 4 0 04 May 2023
State Spaces Aren't Enough: Machine Translation Needs Attention Ali Vardasbi Telmo Pires Robin M. Schmidt Stephan Peitz 24 9 0 25 Apr 2023
Improving Autoregressive NLP Tasks via Modular Linearized Attention Victor Agostinelli Lizhong Chen 24 1 0 17 Apr 2023
Cross Attention Transformers for Multi-modal Unsupervised Whole-Body PET Anomaly Detection Ashay Patel Petru-Daniel Tudosiu W. H. Pinaya G. Cook Vicky Goh Sebastien Ourselin M. Jorge Cardoso OOD ViT MedIm 28 11 0 14 Apr 2023
Scallop: A Language for Neurosymbolic Programming Ziyang Li Jiani Huang Mayur Naik ReLM LRM NAI 24 30 0 10 Apr 2023
Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark Alexander Pan Chan Jun Shern Andy Zou Nathaniel Li Steven Basart Thomas Woodside Jonathan Ng Hanlin Zhang Scott Emmons Dan Hendrycks 26 127 0 06 Apr 2023
Dialogue-Contextualized Re-ranking for Medical History-Taking Jian Zhu Ilya Valmianski Anitha Kannan 19 1 0 04 Apr 2023
CoLT5: Faster Long-Range Transformers with Conditional Computation Joshua Ainslie Tao Lei Michiel de Jong Santiago Ontañón Siddhartha Brahma ... Mandy Guo James Lee-Thorp Yi Tay Yun-hsuan Sung Sumit Sanghai LLMAG 36 63 0 17 Mar 2023
Efficiency 360: Efficient Vision Transformers Badri N. Patro Vijay Srinivas Agneeswaran 26 6 0 16 Feb 2023
Simple Hardware-Efficient Long Convolutions for Sequence Modeling Daniel Y. Fu Elliot L. Epstein Eric N. D. Nguyen A. Thomas Michael Zhang Tri Dao Atri Rudra Christopher Ré 16 52 0 13 Feb 2023
Efficient Attention via Control Variates Lin Zheng Jianbo Yuan Chong-Jun Wang Lingpeng Kong 34 18 0 09 Feb 2023
Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers K. Choromanski Shanda Li Valerii Likhosherstov Kumar Avinava Dubey Shengjie Luo Di He Yiming Yang Tamás Sarlós Thomas Weingarten Adrian Weller 37 8 0 03 Feb 2023
Scaling Up Computer Vision Neural Networks Using Fast Fourier Transform Siddharth Agrawal 15 0 0 02 Feb 2023
Mnemosyne: Learning to Train Transformers with Transformers Deepali Jain K. Choromanski Kumar Avinava Dubey Sumeet Singh Vikas Sindhwani Tingnan Zhang Jie Tan OffRL 39 9 0 02 Feb 2023
Byte Pair Encoding for Symbolic Music Nathan Fradet Nicolas Gutowski F. Chhel Jean-Pierre Briot 29 15 0 27 Jan 2023
Hungry Hungry Hippos: Towards Language Modeling with State Space Models Daniel Y. Fu Tri Dao Khaled Kamal Saab A. Thomas Atri Rudra Christopher Ré 73 370 0 28 Dec 2022
Pretraining Without Attention Junxiong Wang J. Yan Albert Gu Alexander M. Rush 27 48 0 20 Dec 2022
JEMMA: An Extensible Java Dataset for ML4Code Applications Anjan Karmakar Miltiadis Allamanis Romain Robbes VLM 29 3 0 18 Dec 2022
Efficient Long Sequence Modeling via State Space Augmented Transformer Simiao Zuo Xiaodong Liu Jian Jiao Denis Xavier Charles Eren Manavoglu Tuo Zhao Jianfeng Gao 125 36 0 15 Dec 2022
BudgetLongformer: Can we Cheaply Pretrain a SotA Legal Language Model From Scratch? Joel Niklaus Daniele Giofré 33 11 0 30 Nov 2022
Token Turing Machines Michael S. Ryoo K. Gopalakrishnan Kumara Kahatapitiya Ted Xiao Kanishka Rao Austin Stone Yao Lu Julian Ibarz Anurag Arnab 27 21 0 16 Nov 2022
Transformers meet Stochastic Block Models: Attention with Data-Adaptive Sparsity and Cost Sungjun Cho Seonwoo Min Jinwoo Kim Moontae Lee Honglak Lee Seunghoon Hong 40 3 0 27 Oct 2022
The Devil in Linear Transformer Zhen Qin Xiaodong Han Weixuan Sun Dongxu Li Lingpeng Kong Nick Barnes Yiran Zhong 36 70 0 19 Oct 2022
Neural Attentive Circuits Nasim Rahaman M. Weiß Francesco Locatello C. Pal Yoshua Bengio Bernhard Schölkopf Erran L. Li Nicolas Ballas 32 6 0 14 Oct 2022
CAB: Comprehensive Attention Benchmarking on Long Sequence Modeling Jinchao Zhang Shuyang Jiang Jiangtao Feng Lin Zheng Lingpeng Kong 3DV 43 9 0 14 Oct 2022
An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classification Ilias Chalkidis Xiang Dai Manos Fergadiotis Prodromos Malakasiotis Desmond Elliott 34 33 0 11 Oct 2022
WavSpA: Wavelet Space Attention for Boosting Transformers' Long Sequence Learning Ability Yufan Zhuang Zihan Wang Fangbo Tao Jingbo Shang ViT AI4TS 35 3 0 05 Oct 2022
Transformer Meets Boundary Value Inverse Problems Ruchi Guo Shuhao Cao Long Chen MedIm 36 21 0 29 Sep 2022
Liquid Structural State-Space Models Ramin Hasani Mathias Lechner Tsun-Hsuan Wang Makram Chahine Alexander Amini Daniela Rus AI4TS 107 96 0 26 Sep 2022
Fast-FNet: Accelerating Transformer Encoder Models via Efficient Fourier Layers Nurullah Sevim Ege Ozan Özyedek Furkan Şahinuç Aykut Koç 35 11 0 26 Sep 2022
Mega: Moving Average Equipped Gated Attention Xuezhe Ma Chunting Zhou Xiang Kong Junxian He Liangke Gui Graham Neubig Jonathan May Luke Zettlemoyer 33 183 0 21 Sep 2022
Adaptable Butterfly Accelerator for Attention-based NNs via Hardware and Algorithm Co-design Hongxiang Fan Thomas C. P. Chau Stylianos I. Venieris Royson Lee Alexandros Kouris Wayne Luk Nicholas D. Lane Mohamed S. Abdelfattah 37 57 0 20 Sep 2022
Efficient Quantized Sparse Matrix Operations on Tensor Cores Shigang Li Kazuki Osawa Torsten Hoefler 82 31 0 14 Sep 2022
Unified Fully and Timestamp Supervised Temporal Action Segmentation via Sequence to Sequence Translation Nadine Behrmann S. Golestaneh Zico Kolter Juergen Gall M. Noroozi 22 72 0 01 Sep 2022
Efficient Methods for Natural Language Processing: A Survey Marcos Vinícius Treviso Ji-Ung Lee Tianchu Ji Betty van Aken Qingqing Cao ... Emma Strubell Niranjan Balasubramanian Leon Derczynski Iryna Gurevych Roy Schwartz 30 109 0 31 Aug 2022
SpanDrop: Simple and Effective Counterfactual Learning for Long Sequences Peng Qi Guangtao Wang Jing Huang 24 0 0 03 Aug 2022
Efficient Long-Text Understanding with Short-Text Models Maor Ivgi Uri Shaham Jonathan Berant VLM 27 75 0 01 Aug 2022
Momentum Transformer: Closing the Performance Gap Between Self-attention and Its Linearization T. Nguyen Richard G. Baraniuk Robert M. Kirby Stanley J. Osher Bao Wang 32 9 0 01 Aug 2022