Self-attention Does Not Need $O(n^2)$ Memory

10 December 2021

Papers citing "Self-attention Does Not Need $O(n^2)$ Memory"

34 / 34 papers shown

Title
Parallel Sequence Modeling via Generalized Spatial Propagation Network Hongjun Wang Wonmin Byeon Jiarui Xu Liang Feng Ka Chun Cheung Xiaolong Wang Kai Han Jan Kautz Sifei Liu 167 0 0 21 Jan 2025
AsymRnR: Video Diffusion Transformers Acceleration with Asymmetric Reduction and Restoration Wenhao Sun Rong-Cheng Tu Jingyi Liao Zhao Jin Dacheng Tao VGen 102 1 0 16 Dec 2024
Rethinking Transformer for Long Contextual Histopathology Whole Slide Image Analysis Honglin Li Yunlong Zhang Pingyi Chen Zhongyi Shui Chenglu Zhu Lin Yang MedIm 44 4 0 18 Oct 2024
FlashMask: Efficient and Rich Mask Extension of FlashAttention Guoxia Wang Jinle Zeng Xiyuan Xiao Siming Wu Jiabin Yang Lujing Zheng Zeyu Chen Jiang Bian Dianhai Yu Haifeng Wang 157 2 0 02 Oct 2024
Characterizing and Efficiently Accelerating Multimodal Generation Model Inference Yejin Lee Anna Y. Sun Basil Hosmer Bilge Acun Can Balioglu ... Ram Pasunuru Scott Yih Sravya Popuri Xing Liu Carole-Jean Wu 54 2 0 30 Sep 2024
Training Ultra Long Context Language Model with Fully Pipelined Distributed Transformer Jinghan Yao Sam Ade Jacobs Masahiro Tanaka Olatunji Ruwase Hari Subramoni D. Panda 33 2 0 30 Aug 2024
Unsupervised Representation Learning by Balanced Self Attention Matching Daniel Shalam Simon Korman SSL 38 0 0 04 Aug 2024
CatLIP: CLIP-level Visual Recognition Accuracy with 2.7x Faster Pre-training on Web-scale Image-Text Data Sachin Mehta Maxwell Horton Fartash Faghri Mohammad Hossein Sekhavat Mahyar Najibi Mehrdad Farajtabar Oncel Tuzel Mohammad Rastegari VLM CLIP 44 6 0 24 Apr 2024
Linear Attention Sequence Parallelism Weigao Sun Zhen Qin Dong Li Xuyang Shen Yu Qiao Yiran Zhong 73 2 0 03 Apr 2024
Tracking Meets LoRA: Faster Training, Larger Model, Stronger Performance Liting Lin Heng Fan Zhipeng Zhang Yaowei Wang Yong-mei Xu Haibin Ling 49 26 0 08 Mar 2024
RelayAttention for Efficient Large Language Model Serving with Long System Prompts Lei Zhu Xinjiang Wang Wayne Zhang Rynson W. H. Lau 33 6 0 22 Feb 2024
World Model on Million-Length Video And Language With Blockwise RingAttention Hao Liu Wilson Yan Matei A. Zaharia Pieter Abbeel VGen 31 62 0 13 Feb 2024
AutoChunk: Automated Activation Chunk for Memory-Efficient Long Sequence Inference Xuanlei Zhao Shenggan Cheng Guangyang Lu Jiarui Fang Hao Zhou Bin Jia Ziming Liu Yang You MQ 17 3 0 19 Jan 2024
MAMI: Multi-Attentional Mutual-Information for Long Sequence Neuron Captioning Alfirsa Damasyifa Fauzulhaq Wahyu Parwitayasa Joseph A. Sugihdharma M. F. Ridhani N. Yudistira 34 0 0 05 Jan 2024
Long-MIL: Scaling Long Contextual Multiple Instance Learning for Histopathology Whole Slide Image Analysis Honglin Li Yunlong Zhang Chenglu Zhu Jiatong Cai Sunyi Zheng Lin Yang VLM 35 4 0 21 Nov 2023
RoFormer for Position Aware Multiple Instance Learning in Whole Slide Image Classification Etienne Pochet Rami Maroun Roger Trullo MedIm 23 2 0 03 Oct 2023
Transformer-VQ: Linear-Time Transformers via Vector Quantization Albert Mohwald 31 15 0 28 Sep 2023
Baichuan 2: Open Large-scale Language Models Ai Ming Yang Bin Xiao Bingning Wang Borong Zhang Ce Bian ... Youxin Jiang Yuchen Gao Yupeng Zhang Zenan Zhou Zhiying Wu ELM LRM 68 703 0 19 Sep 2023
Memory Efficient Neural Processes via Constant Memory Attention Block Leo Feng Frederick Tung Hossein Hajimirsadeghi Yoshua Bengio Mohamed Osama Ahmed 31 5 0 23 May 2023
RWKV: Reinventing RNNs for the Transformer Era Bo Peng Eric Alcaide Quentin G. Anthony Alon Albalak Samuel Arcadinho ... Qihang Zhao P. Zhou Qinghua Zhou Jian Zhu Rui-Jie Zhu 90 562 0 22 May 2023
ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities Peng Wang Shijie Wang Junyang Lin Shuai Bai Xiaohuan Zhou Jingren Zhou Xinggang Wang Chang Zhou VLM MLLM ObjD 48 115 0 18 May 2023
A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Training Nitay Calderon Subhabrata Mukherjee Roi Reichart Amir Kantor 38 17 0 03 May 2023
On Efficient Training of Large-Scale Deep Learning Models: A Literature Review Li Shen Yan Sun Zhiyuan Yu Liang Ding Xinmei Tian Dacheng Tao VLM 30 41 0 07 Apr 2023
LLaMA: Open and Efficient Foundation Language Models Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux ... Faisal Azhar Aurelien Rodriguez Armand Joulin Edouard Grave Guillaume Lample ALM PILM 34 12,368 0 27 Feb 2023
Efficient Attention via Control Variates Lin Zheng Jianbo Yuan Chong-Jun Wang Lingpeng Kong 34 18 0 09 Feb 2023
Hungry Hungry Hippos: Towards Language Modeling with State Space Models Daniel Y. Fu Tri Dao Khaled Kamal Saab A. Thomas Atri Rudra Christopher Ré 73 370 0 28 Dec 2022
Memory Safe Computations with XLA Compiler A. Artemev Tilman Roeder Mark van der Wilk 29 8 0 28 Jun 2022
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao Daniel Y. Fu Stefano Ermon Atri Rudra Christopher Ré VLM 78 2,045 0 27 May 2022
General-purpose, long-context autoregressive modeling with Perceiver AR Curtis Hawthorne Andrew Jaegle Cătălina Cangea Sebastian Borgeaud C. Nash ... Hannah R. Sheahan Neil Zeghidour Jean-Baptiste Alayrac João Carreira Jesse Engel 43 65 0 15 Feb 2022
FLAT: An Optimized Dataflow for Mitigating Attention Bottlenecks Sheng-Chun Kao Suvinay Subramanian Gaurav Agrawal Amir Yazdanbakhsh T. Krishna 38 57 0 13 Jul 2021
Combiner: Full Attention Transformer with Sparse Computation Cost Hongyu Ren H. Dai Zihang Dai Mengjiao Yang J. Leskovec Dale Schuurmans Bo Dai 81 77 0 12 Jul 2021
Efficient Transformers: A Survey Yi Tay Mostafa Dehghani Dara Bahri Donald Metzler VLM 109 1,102 0 14 Sep 2020
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 285 2,017 0 28 Jul 2020
Efficient Content-Based Sparse Attention with Routing Transformers Aurko Roy M. Saffar Ashish Vaswani David Grangier MoE 252 580 0 12 Mar 2020

Self-attention Does Not Need O(n2)O(n^2)O(n2) Memory

Papers citing "Self-attention Does Not Need $O(n^2)$ Memory"

Self-attention Does Not Need $O(n^2)$ Memory