Transformer Quality in Linear Time

21 February 2022

Papers citing "Transformer Quality in Linear Time"

39 / 39 papers shown

Title
Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free Zihan Qiu Zekun Wang Bo Zheng Zeyu Huang Kaiyue Wen ... Fei Huang Suozhi Huang Dayiheng Liu Jingren Zhou Junyang Lin MoE 28 0 0 10 May 2025
PRE-Mamba: A 4D State Space Model for Ultra-High-Frequent Event Camera Deraining Ciyu Ruan Ruishan Guo Zihang Gong J. Xu Wenhan Yang Xinlei Chen Mamba 67 0 0 08 May 2025
Hadamard product in deep learning: Introduction, Advances and Challenges Grigorios G. Chrysos Yongtao Wu Razvan Pascanu Philip Torr V. Cevher AAML 98 0 0 17 Apr 2025
Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kernels M. Beck Korbinian Poppel Phillip Lippe Sepp Hochreiter 61 1 0 18 Mar 2025
Parallel Sequence Modeling via Generalized Spatial Propagation Network Hongjun Wang Wonmin Byeon Jiarui Xu Liang Feng Ka Chun Cheung Xiaolong Wang Kai Han Jan Kautz Sifei Liu 152 0 0 21 Jan 2025
Epicardium Prompt-guided Real-time Cardiac Ultrasound Frame-to-volume Registration Long Lei Jun Zhou Jialun Pei Baoliang Zhao Yueming Jin Yuen-Chun Jeremy Teoh Jing Qin Pheng-Ann Heng 64 1 0 20 Jan 2025
HSR-Enhanced Sparse Attention Acceleration Bo Chen Yingyu Liang Zhizhou Sha Zhenmei Shi Zhao-quan Song 93 18 0 14 Oct 2024
Rodimus*: Breaking the Accuracy-Efficiency Trade-Off with Efficient Attentions Zhihao He Hang Yu Zi Gong Shizhan Liu J. Li Weiyao Lin VLM 38 1 0 09 Oct 2024
Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequences Zicheng Liu Siyuan Li Li Wang Zedong Wang Yunfan Liu Stan Z. Li 35 7 0 12 Jun 2024
SFTformer: A Spatial-Frequency-Temporal Correlation-Decoupling Transformer for Radar Echo Extrapolation Liangyu Xu Wanxuan Lu Hongfeng Yu Fanglong Yao Xian Sun Kun Fu 45 5 0 28 Feb 2024
Investigating Recurrent Transformers with Dynamic Halt Jishnu Ray Chowdhury Cornelia Caragea 39 1 0 01 Feb 2024
MIMONets: Multiple-Input-Multiple-Output Neural Networks Exploiting Computation in Superposition Nicolas Menet Michael Hersche G. Karunaratne Luca Benini Abu Sebastian Abbas Rahimi 28 13 0 05 Dec 2023
RETSim: Resilient and Efficient Text Similarity Marina Zhang Owen Vallis Aysegul Bumin Tanay Vakharia Elie Bursztein 28 1 0 28 Nov 2023
Transformer-VQ: Linear-Time Transformers via Vector Quantization Albert Mohwald 28 15 0 28 Sep 2023
Small-scale proxies for large-scale Transformer training instabilities Mitchell Wortsman Peter J. Liu Lechao Xiao Katie Everett A. Alemi ... Jascha Narain Sohl-Dickstein Kelvin Xu Jaehoon Lee Justin Gilmer Simon Kornblith 35 81 0 25 Sep 2023
Incrementally-Computable Neural Networks: Efficient Inference for Dynamic Inputs Or Sharir Anima Anandkumar 29 0 0 27 Jul 2023
FIT: Far-reaching Interleaved Transformers Ting-Li Chen Lala Li 26 12 0 22 May 2023
Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inference Tao Lei Junwen Bai Siddhartha Brahma Joshua Ainslie Kenton Lee ... Vincent Zhao Yuexin Wu Bo-wen Li Yu Zhang Ming-Wei Chang BDL AI4CE 28 54 0 11 Apr 2023
On Efficient Training of Large-Scale Deep Learning Models: A Literature Review Li Shen Yan Sun Zhiyuan Yu Liang Ding Xinmei Tian Dacheng Tao VLM 30 41 0 07 Apr 2023
RTMPose: Real-Time Multi-Person Pose Estimation based on MMPose Tao Jiang Peng Lu Li Zhang Ning Ma Rui Han Chengqi Lyu Yining Li Kai-xiang Chen 3DH 45 158 0 13 Mar 2023
RETVec: Resilient and Efficient Text Vectorizer Elie Bursztein Marina Zhang Owen Vallis Xinyu Jia Alexey Kurakin VLM 29 4 0 18 Feb 2023
Symbolic Discovery of Optimization Algorithms Xiangning Chen Chen Liang Da Huang Esteban Real Kaiyuan Wang ... Xuanyi Dong Thang Luong Cho-Jui Hsieh Yifeng Lu Quoc V. Le 61 350 0 13 Feb 2023
Efficient Attention via Control Variates Lin Zheng Jianbo Yuan Chong-Jun Wang Lingpeng Kong 31 18 0 09 Feb 2023
Pretraining Without Attention Junxiong Wang J. Yan Albert Gu Alexander M. Rush 27 48 0 20 Dec 2022
Efficient Long Sequence Modeling via State Space Augmented Transformer Simiao Zuo Xiaodong Liu Jian Jiao Denis Xavier Charles Eren Manavoglu Tuo Zhao Jianfeng Gao 125 36 0 15 Dec 2022
Meta-Learning Fast Weight Language Models Kevin Clark Kelvin Guu Ming-Wei Chang Panupong Pasupat Geoffrey E. Hinton Mohammad Norouzi KELM 32 13 0 05 Dec 2022
The Devil in Linear Transformer Zhen Qin Xiaodong Han Weixuan Sun Dongxu Li Lingpeng Kong Nick Barnes Yiran Zhong 36 70 0 19 Oct 2022
CAB: Comprehensive Attention Benchmarking on Long Sequence Modeling Jinchao Zhang Shuyang Jiang Jiangtao Feng Lin Zheng Lingpeng Kong 3DV 43 9 0 14 Oct 2022
Multi-scale Attention Network for Single Image Super-Resolution Yan Wang Yusen Li Gang Wang Xiaoguang Liu SupR 39 37 0 28 Sep 2022
Mega: Moving Average Equipped Gated Attention Xuezhe Ma Chunting Zhou Xiang Kong Junxian He Liangke Gui Graham Neubig Jonathan May Luke Zettlemoyer 14 183 0 21 Sep 2022
QSAN: A Near-term Achievable Quantum Self-Attention Network Jinjing Shi Ren-Xin Zhao Wenxuan Wang Shenmin Zhang Xuelong Li 21 20 0 14 Jul 2022
Long Range Language Modeling via Gated State Spaces Harsh Mehta Ankit Gupta Ashok Cutkosky Behnam Neyshabur Mamba 31 231 0 27 Jun 2022
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao Daniel Y. Fu Stefano Ermon Atri Rudra Christopher Ré VLM 63 2,024 0 27 May 2022
Simple Baselines for Image Restoration Liangyu Chen Xiaojie Chu Xinming Zhang Jian Sun 53 833 0 10 Apr 2022
Block-Recurrent Transformers DeLesley S. Hutchins Imanol Schlag Yuhuai Wu Ethan Dyer Behnam Neyshabur 20 94 0 11 Mar 2022
Primer: Searching for Efficient Transformers for Language Modeling David R. So Wojciech Mañke Hanxiao Liu Zihang Dai Noam M. Shazeer Quoc V. Le VLM 85 152 0 17 Sep 2021
Combiner: Full Attention Transformer with Sparse Computation Cost Hongyu Ren H. Dai Zihang Dai Mengjiao Yang J. Leskovec Dale Schuurmans Bo Dai 76 77 0 12 Jul 2021
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 280 2,015 0 28 Jul 2020
Efficient Content-Based Sparse Attention with Routing Transformers Aurko Roy M. Saffar Ashish Vaswani David Grangier MoE 243 580 0 12 Mar 2020