Linformer: Self-Attention with Linear Complexity

8 June 2020

Sinong Wang

Belinda Z. Li

Madian Khabsa

Han Fang

Hao Ma

ArXiv PDF HTML

Papers citing "Linformer: Self-Attention with Linear Complexity"

50 / 1,050 papers shown

Title
Conditional Permutation Invariant Flows Berend Zwartsenberg Adam Scibior Matthew Niedoba Vasileios Lioutas Yunpeng Liu Justice Sefas Setareh Dabiri J. Lavington Trevor Campbell Frank Wood 17 8 0 17 Jun 2022
CMT-DeepLab: Clustering Mask Transformers for Panoptic Segmentation Qihang Yu Huiyu Wang Dahun Kim Siyuan Qiao Maxwell D. Collins Yukun Zhu Hartwig Adam Alan Yuille Liang-Chieh Chen ViT MedIm 37 90 0 17 Jun 2022
SimA: Simple Softmax-free Attention for Vision Transformers Soroush Abbasi Koohpayegani Hamed Pirsiavash 26 25 0 17 Jun 2022
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks Tuan Dinh Yuchen Zeng Ruisu Zhang Ziqian Lin Michael Gira Shashank Rajput Jy-yong Sohn Dimitris Papailiopoulos Kangwook Lee LMTD 52 128 0 14 Jun 2022
MMMNA-Net for Overall Survival Time Prediction of Brain Tumor Patients Wen Tang Haoyue Zhang Pengxin Yu Han Kang Rongguo Zhang 40 6 0 13 Jun 2022
ChordMixer: A Scalable Neural Attention Model for Sequences with Different Lengths Ruslan Khalitov Tong Yu Lei Cheng Zhirong Yang 33 12 0 12 Jun 2022
Bootstrapping Multi-view Representations for Fake News Detection Qichao Ying Xiaoxiao Hu Yangming Zhou Zhenxing Qian Dan Zeng Shiming Ge 29 45 0 12 Jun 2022
GateHUB: Gated History Unit with Background Suppression for Online Action Detection Junwen Chen Gaurav Mittal Ye Yu Yu Kong Mei Chen 52 33 0 09 Jun 2022
SparseFormer: Attention-based Depth Completion Network Frederik Warburg Michael Ramamonjisoa Manuel López-Antequera MoE MDE 29 4 0 09 Jun 2022
Separable Self-attention for Mobile Vision Transformers Sachin Mehta Mohammad Rastegari ViT MQ 34 253 0 06 Jun 2022
DeeprETA: An ETA Post-processing System at Scale Xinyu Hu Tanmay Binaykiya Eric C. Frank Olcay Cirit 33 13 0 05 Jun 2022
EAANet: Efficient Attention Augmented Convolutional Networks Runqing Zhang Tianshu Zhu 30 0 0 03 Jun 2022
Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectives Jun Li Junyu Chen Yucheng Tang Ce Wang Bennett A. Landman S. K. Zhou ViT OOD MedIm 28 24 0 02 Jun 2022
BayesFormer: Transformer with Uncertainty Estimation Karthik Abinav Sankararaman Sinong Wang Han Fang UQCV BDL 30 10 0 02 Jun 2022
Dynamic Linear Transformer for 3D Biomedical Image Segmentation Zheyu Zhang Ulas Bagci ViT MedIm 36 12 0 01 Jun 2022
Fair Comparison between Efficient Attentions Jiuk Hong Chaehyeon Lee Soyoun Bang Heechul Jung 28 1 0 01 Jun 2022
Transformer with Fourier Integral Attentions T. Nguyen Minh Pham Tam Nguyen Khai Nguyen Stanley J. Osher Nhat Ho 32 4 0 01 Jun 2022
Transformers for Multi-Object Tracking on Point Clouds Felicia Ruppel F. Faion Claudius Gläser Klaus C. J. Dietmayer 3DPC 26 17 0 31 May 2022
Chefs' Random Tables: Non-Trigonometric Random Features Valerii Likhosherstov K. Choromanski Kumar Avinava Dubey Frederick Liu Tamás Sarlós Adrian Weller 38 17 0 30 May 2022
Temporal Latent Bottleneck: Synthesis of Fast and Slow Processing Mechanisms in Sequence Learning Aniket Didolkar Kshitij Gupta Anirudh Goyal Nitesh B. Gundavarapu Alex Lamb Nan Rosemary Ke Yoshua Bengio AI4CE 126 17 0 30 May 2022
EfficientViT: Multi-Scale Linear Attention for High-Resolution Dense Prediction Han Cai Junyan Li Muyan Hu Chuang Gan Song Han 37 49 0 29 May 2022
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao Daniel Y. Fu Stefano Ermon Atri Rudra Christopher Ré VLM 116 2,061 0 27 May 2022
Future Transformer for Long-term Action Anticipation Dayoung Gong Joonseok Lee Manjin Kim S. Ha Minsu Cho AI4TS 16 62 0 27 May 2022
X-ViT: High Performance Linear Vision Transformer without Softmax Jeonggeun Song Heung-Chang Lee ViT 33 2 0 27 May 2022
Fast Vision Transformers with HiLo Attention Zizheng Pan Jianfei Cai Bohan Zhuang 28 152 0 26 May 2022
Training Language Models with Memory Augmentation Zexuan Zhong Tao Lei Danqi Chen RALM 249 128 0 25 May 2022
Leveraging Locality in Abstractive Text Summarization Yixin Liu Ansong Ni Linyong Nan Budhaditya Deb Chenguang Zhu Ahmed Hassan Awadallah Dragomir R. Radev 35 18 0 25 May 2022
Recipe for a General, Powerful, Scalable Graph Transformer Ladislav Rampášek Mikhail Galkin Vijay Prakash Dwivedi Anh Tuan Luu Guy Wolf Dominique Beaini 80 527 0 25 May 2022
History Compression via Language Models in Reinforcement Learning Fabian Paischer Thomas Adler Vihang Patil Angela Bitto-Nemling Markus Holzleitner Sebastian Lehner Hamid Eghbalzadeh Sepp Hochreiter OffRL AI4TS 30 42 0 24 May 2022
ASSET: Autoregressive Semantic Scene Editing with Transformers at High Resolutions Difan Liu Sandesh Shetty Tobias Hinz Matthew Fisher Richard Y. Zhang Taesung Park E. Kalogerakis ViT 32 30 0 24 May 2022
Semi-Parametric Inducing Point Networks and Neural Processes R. Rastogi Yair Schiff Alon Hacohen Zhaozhi Li I-Hsiang Lee Yuntian Deng M. Sabuncu Volodymyr Kuleshov 3DPC 29 6 0 24 May 2022
TransforMatcher: Match-to-Match Attention for Semantic Correspondence Seungwook Kim Juhong Min Minsu Cho ViT 51 32 0 23 May 2022
Dynamic Query Selection for Fast Visual Perceiver Corentin Dancette Matthieu Cord 36 1 0 22 May 2022
Sampling Is All You Need on Modeling Long-Term User Behaviors for CTR Prediction Yue Cao Xiaojiang Zhou Jiaqi Feng Peihao Huang Yao Xiao Dayao Chen Sheng Chen 82 40 0 20 May 2022
Exploring Extreme Parameter Compression for Pre-trained Language Models Yuxin Ren Benyou Wang Lifeng Shang Xin Jiang Qun Liu 33 18 0 20 May 2022
Towards Unified Keyframe Propagation Models Patrick Esser Peter Michael Soumyadip Sengupta VGen 35 0 0 19 May 2022
FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecasting Tian Zhou Ziqing Ma Xue Wang Qingsong Wen Liang Sun Tao Yao Wotao Yin Rong Jin AI4TS 121 171 0 18 May 2022
Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformers Arda Sahiner Tolga Ergen Batu Mehmet Ozturkler John M. Pauly Morteza Mardani Mert Pilanci 43 33 0 17 May 2022
ShiftAddNAS: Hardware-Inspired Search for More Accurate and Efficient Neural Networks Haoran You Baopu Li Huihong Shi Y. Fu Yingyan Lin 52 17 0 17 May 2022
Multiformer: A Head-Configurable Transformer-Based Model for Direct Speech Translation Gerard Sant Gerard I. Gállego Belen Alastruey Marta R. Costa-jussá 22 3 0 14 May 2022
Vision Transformer: Vit and its Derivatives Zujun Fu ViT 41 6 0 12 May 2022
Reduce Information Loss in Transformers for Pluralistic Image Inpainting Qiankun Liu Zhentao Tan Dongdong Chen Qi Chu Xiyang Dai Yinpeng Chen Mengchen Liu Lu Yuan Nenghai Yu ViT 31 70 0 10 May 2022
Task-specific Compression for Multi-task Language Models using Attribution-based Pruning Nakyeong Yang Yunah Jang Hwanhee Lee Seohyeong Jung Kyomin Jung 24 8 0 09 May 2022
EdgeViTs: Competing Light-weight CNNs on Mobile Devices with Vision Transformers Junting Pan Adrian Bulat Fuwen Tan Xiatian Zhu Łukasz Dudziak Hongsheng Li Georgios Tzimiropoulos Brais Martínez ViT 36 183 0 06 May 2022
CenterCLIP: Token Clustering for Efficient Text-Video Retrieval Shuai Zhao Linchao Zhu Xiaohan Wang Yi Yang VLM CLIP 20 112 0 02 May 2022
LayoutBERT: Masked Language Layout Model for Object Insertion Kerem Turgutlu Sanatan Sharma J. Kumar VLM DiffM 38 2 0 30 Apr 2022
Depth Estimation with Simplified Transformer John Yang Le An Anurag Dixit Jinkyu Koo Su Inn Park MDE 41 21 0 28 Apr 2022
Triformer: Triangular, Variable-Specific Attentions for Long Sequence Multivariate Time Series Forecasting--Full Version Razvan-Gabriel Cirstea Chenjuan Guo B. Yang Tung Kieu Xuanyi Dong Shirui Pan AI4TS 37 108 0 28 Apr 2022
Towards Flexible Inference in Sequential Decision Problems via Bidirectional Transformers Micah Carroll Jessy Lin Orr Paradise Raluca Georgescu Mingfei Sun ... Stephanie Milani Katja Hofmann Matthew J. Hausknecht Anca Dragan Sam Devlin OffRL 45 10 0 28 Apr 2022
A survey on attention mechanisms for medical applications: are we moving towards better algorithms? Tiago Gonçalves Isabel Rio-Torto Luís F. Teixeira J. S. Cardoso OOD MedIm 42 36 0 26 Apr 2022