Grouped self-attention mechanism for a memory-efficient Transformer

Grouped self-attention mechanism for a memory-efficient Transformer

2 October 2022

Papers citing "Grouped self-attention mechanism for a memory-efficient Transformer"

12 / 12 papers shown

Title
Twins: Revisiting the Design of Spatial Attention in Vision Transformers Xiangxiang Chu Zhi Tian Yuqing Wang Bo Zhang Haibing Ren Xiaolin K. Wei Huaxia Xia Chunhua Shen ViT 79 1,017 0 28 Apr 2021
Rethinking Attention with Performers K. Choromanski Valerii Likhosherstov David Dohan Xingyou Song Andreea Gane ... Afroz Mohiuddin Lukasz Kaiser David Belanger Lucy J. Colwell Adrian Weller 169 1,577 0 30 Sep 2020
Linformer: Self-Attention with Linear Complexity Sinong Wang Belinda Z. Li Madian Khabsa Han Fang Hao Ma 193 1,700 0 08 Jun 2020
Longformer: The Long-Document Transformer Iz Beltagy Matthew E. Peters Arman Cohan RALM VLM 138 4,061 0 10 Apr 2020
Reformer: The Efficient Transformer Nikita Kitaev Lukasz Kaiser Anselm Levskaya VLM 182 2,311 0 13 Jan 2020
Compressive Transformers for Long-Range Sequence Modelling Jack W. Rae Anna Potapenko Siddhant M. Jayakumar Timothy Lillicrap RALM VLM KELM 64 646 0 13 Nov 2019
Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting Shiyang Li Xiaoyong Jin Yao Xuan Xiyou Zhou Wenhu Chen Yu Wang Xifeng Yan AI4TS 85 1,414 0 29 Jun 2019
Generating Long Sequences with Sparse Transformers R. Child Scott Gray Alec Radford Ilya Sutskever 102 1,896 0 23 Apr 2019
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context Zihang Dai Zhilin Yang Yiming Yang J. Carbonell Quoc V. Le Ruslan Salakhutdinov VLM 214 3,724 0 09 Jan 2019
Deep Learning with Long Short-Term Memory for Time Series Prediction Yuxiu Hua Zhifeng Zhao Rongpeng Li Xianfu Chen Zhiming Liu Honggang Zhang AI4TS 47 428 0 24 Oct 2018
Image Transformer Niki Parmar Ashish Vaswani Jakob Uszkoreit Lukasz Kaiser Noam M. Shazeer Alexander Ku Dustin Tran ViT 128 1,679 0 15 Feb 2018
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 665 131,414 0 12 Jun 2017