v1v2 (latest)

Transformer Quality in Linear Time

21 February 2022

Papers citing "Transformer Quality in Linear Time"

42 / 42 papers shown

Title
PRE-Mamba: A 4D State Space Model for Ultra-High-Frequent Event Camera Deraining Ciyu Ruan Ruishan Guo Zihang Gong Jinfeng Xu Wenhan Yang Xinlei Chen Mamba 104 0 0 08 May 2025
Reducing Smoothness with Expressive Memory Enhanced Hierarchical Graph Neural Networks Thomas Bailie Yun Sing Koh S. Karthik Mukkavilli V. Vetrova AI4TS 236 0 0 01 Apr 2025
Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kernels M. Beck Korbinian Poppel Phillip Lippe Sepp Hochreiter 150 4 0 18 Mar 2025
Epicardium Prompt-guided Real-time Cardiac Ultrasound Frame-to-volume Registration Long Lei Jun Zhou Jialun Pei Baoliang Zhao Yueming Jin Yuen-Chun Jeremy Teoh Jing Qin Pheng-Ann Heng 222 2 0 20 Jan 2025
HSR-Enhanced Sparse Attention Acceleration Bo Chen Yingyu Liang Zhizhou Sha Zhenmei Shi Zhao Song 242 22 0 14 Oct 2024
Investigating Recurrent Transformers with Dynamic Halt Jishnu Ray Chowdhury Cornelia Caragea 171 1 0 01 Feb 2024
CAB: Comprehensive Attention Benchmarking on Long Sequence Modeling Jinchao Zhang Shuyang Jiang Jiangtao Feng Lin Zheng Dianbo Sui 3DV 165 9 0 14 Oct 2022
LaMDA: Language Models for Dialog Applications R. Thoppilan Daniel De Freitas Jamie Hall Noam M. Shazeer Apoorv Kulshreshtha ... Blaise Aguera-Arcas Claire Cui M. Croak Ed H. Chi Quoc Le ALM 146 1,602 0 20 Jan 2022
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts Nan Du Yanping Huang Andrew M. Dai Simon Tong Dmitry Lepikhin ... Kun Zhang Quoc V. Le Yonghui Wu Zhiwen Chen Claire Cui ALM MoE 246 829 0 13 Dec 2021
Primer: Searching for Efficient Transformers for Language Modeling David R. So Wojciech Mañke Hanxiao Liu Zihang Dai Noam M. Shazeer Quoc V. Le VLM 264 156 0 17 Sep 2021
Combiner: Full Attention Transformer with Sparse Computation Cost Hongyu Ren H. Dai Zihang Dai Mengjiao Yang J. Leskovec Dale Schuurmans Bo Dai 156 80 0 12 Jul 2021
Pay Attention to MLPs Hanxiao Liu Zihang Dai David R. So Quoc V. Le AI4CE 135 664 0 17 May 2021
RoFormer: Enhanced Transformer with Rotary Position Embedding Jianlin Su Yu Lu Shengfeng Pan Ahmed Murtadha Bo Wen Yunfeng Liu 329 2,533 0 20 Apr 2021
Perceiver: General Perception with Iterative Attention Andrew Jaegle Felix Gimeno Andrew Brock Andrew Zisserman Oriol Vinyals João Carreira VLM ViT MDE 212 1,024 0 04 Mar 2021
Random Feature Attention Hao Peng Nikolaos Pappas Dani Yogatama Roy Schwartz Noah A. Smith Lingpeng Kong 115 362 0 03 Mar 2021
Do Transformer Modifications Transfer Across Implementations and Applications? Sharan Narang Hyung Won Chung Yi Tay W. Fedus Thibault Févry ... Wei Li Nan Ding Jake Marcus Adam Roberts Colin Raffel 92 127 0 23 Feb 2021
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai ... Matthias Minderer G. Heigold Sylvain Gelly Jakob Uszkoreit N. Houlsby ViT 684 41,563 0 22 Oct 2020
Rethinking Attention with Performers K. Choromanski Valerii Likhosherstov David Dohan Xingyou Song Andreea Gane ... Afroz Mohiuddin Lukasz Kaiser David Belanger Lucy J. Colwell Adrian Weller 186 1,604 0 30 Sep 2020
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 582 2,105 0 28 Jul 2020
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention Angelos Katharopoulos Apoorv Vyas Nikolaos Pappas Franccois Fleuret 209 1,793 0 29 Jun 2020
Linformer: Self-Attention with Linear Complexity Sinong Wang Belinda Z. Li Madian Khabsa Han Fang Hao Ma 219 1,719 0 08 Jun 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 908 42,520 0 28 May 2020
Longformer: The Long-Document Transformer Iz Beltagy Matthew E. Peters Arman Cohan RALM VLM 187 4,105 0 10 Apr 2020
Efficient Content-Based Sparse Attention with Routing Transformers Aurko Roy M. Saffar Ashish Vaswani David Grangier MoE 362 604 0 12 Mar 2020
GLU Variants Improve Transformer Noam M. Shazeer 156 1,024 0 12 Feb 2020
Reformer: The Efficient Transformer Nikita Kitaev Lukasz Kaiser Anselm Levskaya VLM 209 2,335 0 13 Jan 2020
Axial Attention in Multidimensional Transformers Jonathan Ho Nal Kalchbrenner Dirk Weissenborn Tim Salimans 115 533 0 20 Dec 2019
Compressive Transformers for Long-Range Sequence Modelling Jack W. Rae Anna Potapenko Siddhant M. Jayakumar Timothy Lillicrap RALM VLM KELM 77 655 0 13 Nov 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 515 20,376 0 23 Oct 2019
Transformers without Tears: Improving the Normalization of Self-Attention Toan Q. Nguyen Julian Salazar 94 231 0 14 Oct 2019
Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting Shiyang Li Xiaoyong Jin Yao Xuan Xiyou Zhou Wenhu Chen Yu Wang Xifeng Yan AI4TS 127 1,429 0 29 Jun 2019
Generating Long Sequences with Sparse Transformers R. Child Scott Gray Alec Radford Ilya Sutskever 136 1,919 0 23 Apr 2019
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context Zihang Dai Zhilin Yang Yiming Yang J. Carbonell Quoc V. Le Ruslan Salakhutdinov VLM 263 3,747 0 09 Jan 2019
CCNet: Criss-Cross Attention for Semantic Segmentation Zilong Huang Xinggang Wang Yunchao Wei Lichao Huang Humphrey Shi Wenyu Liu Chang Huang VOS 214 2,554 0 28 Nov 2018
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 1.8K 95,324 0 11 Oct 2018
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 814 132,725 0 12 Jun 2017
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension Mandar Joshi Eunsol Choi Daniel S. Weld Luke Zettlemoyer RALM 248 2,692 0 09 May 2017
Convolutional Sequence to Sequence Learning Jonas Gehring Michael Auli David Grangier Denis Yarats Yann N. Dauphin AIMat 174 3,292 0 08 May 2017
Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning Stefan Elfwing E. Uchibe Kenji Doya 145 1,746 0 10 Feb 2017
Language Modeling with Gated Convolutional Networks Yann N. Dauphin Angela Fan Michael Auli David Grangier 247 2,410 0 23 Dec 2016
Layer Normalization Jimmy Lei Ba J. Kiros Geoffrey E. Hinton 437 10,548 0 21 Jul 2016
Gaussian Error Linear Units (GELUs) Dan Hendrycks Kevin Gimpel 178 5,049 0 27 Jun 2016