Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention

7 February 2021

Mingxing Tan

Yin Li

Papers citing "Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention"

50 / 116 papers shown

Title
Continuous Spatiotemporal Transformers Antonio H. O. Fonseca E. Zappala J. O. Caro David van Dijk 26 7 0 31 Jan 2023
Local Learning on Transformers via Feature Reconstruction P. Pathak Jingwei Zhang Dimitris Samaras ViT 24 5 0 29 Dec 2022
Multi-Scale Relational Graph Convolutional Network for Multiple Instance Learning in Histopathology Images Roozbeh Bazargani L. Fazli L. Goldenberg M. Gleave A. Bashashati Septimiu Salcudean MedIm 36 7 0 17 Dec 2022
Efficient Long Sequence Modeling via State Space Augmented Transformer Simiao Zuo Xiaodong Liu Jian Jiao Denis Xavier Charles Eren Manavoglu Tuo Zhao Jianfeng Gao 125 36 0 15 Dec 2022
Orthogonal SVD Covariance Conditioning and Latent Disentanglement Yue Song N. Sebe Wei Wang 29 6 0 11 Dec 2022
Transformers meet Stochastic Block Models: Attention with Data-Adaptive Sparsity and Cost Sungjun Cho Seonwoo Min Jinwoo Kim Moontae Lee Honglak Lee Seunghoon Hong 40 3 0 27 Oct 2022
Dense but Efficient VideoQA for Intricate Compositional Reasoning Jihyeon Janel Lee Wooyoung Kang Eun-Sol Kim CoGe 19 3 0 19 Oct 2022
CAB: Comprehensive Attention Benchmarking on Long Sequence Modeling Jinchao Zhang Shuyang Jiang Jiangtao Feng Lin Zheng Lingpeng Kong 3DV 43 9 0 14 Oct 2022
WavSpA: Wavelet Space Attention for Boosting Transformers' Long Sequence Learning Ability Yufan Zhuang Zihan Wang Fangbo Tao Jingbo Shang ViT AI4TS 35 3 0 05 Oct 2022
Liquid Structural State-Space Models Ramin Hasani Mathias Lechner Tsun-Hsuan Wang Makram Chahine Alexander Amini Daniela Rus AI4TS 107 96 0 26 Sep 2022
Fast-FNet: Accelerating Transformer Encoder Models via Efficient Fourier Layers Nurullah Sevim Ege Ozan Özyedek Furkan Şahinuç Aykut Koç 35 11 0 26 Sep 2022
Adaptable Butterfly Accelerator for Attention-based NNs via Hardware and Algorithm Co-design Hongxiang Fan Thomas C. P. Chau Stylianos I. Venieris Royson Lee Alexandros Kouris Wayne Luk Nicholas D. Lane Mohamed S. Abdelfattah 40 58 0 20 Sep 2022
Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humans John J. Nay ELM AILaw 88 27 0 14 Sep 2022
A Circular Window-based Cascade Transformer for Online Action Detection Shuyuan Cao Weihua Luo Bairui Wang Wei Emma Zhang Lin Ma 42 6 0 30 Aug 2022
Deep is a Luxury We Don't Have Ahmed Taha Yen Nhi Truong Vu Brent Mombourquette Thomas P. Matthews Jason Su Sadanand Singh ViT MedIm 26 2 0 11 Aug 2022
Momentum Transformer: Closing the Performance Gap Between Self-attention and Its Linearization T. Nguyen Richard G. Baraniuk Robert M. Kirby Stanley J. Osher Bao Wang 32 9 0 01 Aug 2022
Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentation Sunghwan Hong Seokju Cho Jisu Nam Stephen Lin Seung Wook Kim ViT 24 123 0 22 Jul 2022
Conditional DETR V2: Efficient Detection Transformer with Box Queries Xiaokang Chen Fangyun Wei Gang Zeng Jingdong Wang ViT 30 33 0 18 Jul 2022
QSAN: A Near-term Achievable Quantum Self-Attention Network Jinjing Shi Ren-Xin Zhao Wenxuan Wang Shenmin Zhang Xuelong Li 21 20 0 14 Jul 2022
Pure Transformers are Powerful Graph Learners Jinwoo Kim Tien Dat Nguyen Seonwoo Min Sungjun Cho Moontae Lee Honglak Lee Seunghoon Hong 43 189 0 06 Jul 2022
Improving Covariance Conditioning of the SVD Meta-layer by Orthogonality Yue Song N. Sebe Wei Wang 19 8 0 05 Jul 2022
Deformable Graph Transformer Jinyoung Park Seongjun Yun Hyeon-ju Park Jaewoo Kang Jisu Jeong KyungHyun Kim Jung-Woo Ha Hyunwoo J. Kim 90 7 0 29 Jun 2022
SimA: Simple Softmax-free Attention for Vision Transformers Soroush Abbasi Koohpayegani Hamed Pirsiavash 24 25 0 17 Jun 2022
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao Daniel Y. Fu Stefano Ermon Atri Rudra Christopher Ré VLM 78 2,045 0 27 May 2022
Dynamic Query Selection for Fast Visual Perceiver Corentin Dancette Matthieu Cord 36 1 0 22 May 2022
HCFormer: Unified Image Segmentation with Hierarchical Clustering Teppei Suzuki 27 0 0 20 May 2022
KERPLE: Kernelized Relative Positional Embedding for Length Extrapolation Ta-Chung Chi Ting-Han Fan Peter J. Ramadge Alexander I. Rudnicky 47 65 0 20 May 2022
A graph-transformer for whole slide image classification Yi Zheng R. Gindra Emily J. Green E. Burks Margrit Betke J. Beane V. Kolachalama MedIm 45 123 0 19 May 2022
Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformers Arda Sahiner Tolga Ergen Batu Mehmet Ozturkler John M. Pauly Morteza Mardani Mert Pilanci 40 33 0 17 May 2022
EdgeViTs: Competing Light-weight CNNs on Mobile Devices with Vision Transformers Junting Pan Adrian Bulat Fuwen Tan Xiatian Zhu L. Dudziak Hongsheng Li Georgios Tzimiropoulos Brais Martínez ViT 31 181 0 06 May 2022
Paramixer: Parameterizing Mixing Links in Sparse Factors Works Better than Dot-Product Self-Attention Tong Yu Ruslan Khalitov Lei Cheng Zhirong Yang MoE 27 10 0 22 Apr 2022
Linearizing Transformer with Key-Value Memory Yizhe Zhang Deng Cai 22 5 0 23 Mar 2022
DCT-Former: Efficient Self-Attention with Discrete Cosine Transform Carmelo Scribano Giorgia Franchini M. Prato Marko Bertogna 18 21 0 02 Mar 2022
Dynamic N:M Fine-grained Structured Sparse Attention Mechanism Zhaodong Chen Yuying Quan Zheng Qu L. Liu Yufei Ding Yuan Xie 36 22 0 28 Feb 2022
cosFormer: Rethinking Softmax in Attention Zhen Qin Weixuan Sun Huicai Deng Dongxu Li Yunshen Wei Baohong Lv Junjie Yan Lingpeng Kong Yiran Zhong 29 212 0 17 Feb 2022
ActionFormer: Localizing Moments of Actions with Transformers Chen-Da Liu-Zhang Jianxin Wu Yin Li ViT 31 330 0 16 Feb 2022
CATs++: Boosting Cost Aggregation with Convolutions and Transformers Seokju Cho Sunghwan Hong Seung Wook Kim ViT 27 34 0 14 Feb 2022
Flowformer: Linearizing Transformers with Conservation Flows Haixu Wu Jialong Wu Jiehui Xu Jianmin Wang Mingsheng Long 14 90 0 13 Feb 2022
Exploiting Spatial Sparsity for Event Cameras with Visual Transformers Zuowen Wang Yuhuang Hu Shih-Chii Liu ViT 33 33 0 10 Feb 2022
FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting Tian Zhou Ziqing Ma Qingsong Wen Xue Wang Liang Sun Rong Jin AI4TS 30 1,307 0 30 Jan 2022
CsFEVER and CTKFacts: Acquiring Czech data for fact verification Herbert Ullrich Jan Drchal Martin Rýpar Hana Vincourová Václav Moravec HILM 25 9 0 26 Jan 2022
Continual Transformers: Redundancy-Free Attention for Online Inference Lukas Hedegaard Arian Bakhtiarnia Alexandros Iosifidis CLL 27 11 0 17 Jan 2022
Classification of Long Sequential Data using Circular Dilated Convolutional Neural Networks Lei Cheng Ruslan Khalitov Tong Yu Zhirong Yang 25 32 0 06 Jan 2022
Learning Operators with Coupled Attention Georgios Kissas Jacob H. Seidman Leonardo Ferreira Guilhoto V. Preciado George J. Pappas P. Perdikaris 32 110 0 04 Jan 2022
Multi-Dimensional Model Compression of Vision Transformer Zejiang Hou S. Kung ViT 25 16 0 31 Dec 2021
Couplformer:Rethinking Vision Transformer with Coupling Attention Map Hai Lan Xihao Wang Xian Wei ViT 34 3 0 10 Dec 2021
Forward Operator Estimation in Generative Models with Kernel Transfer Operators Z. Huang Rudrasis Chakraborty Vikas Singh GAN 14 3 0 01 Dec 2021
Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models Tri Dao Beidi Chen Kaizhao Liang Jiaming Yang Zhao Song Atri Rudra Christopher Ré 33 75 0 30 Nov 2021
SOFT: Softmax-free Transformer with Linear Complexity Jiachen Lu Jinghan Yao Junge Zhang Martin Danelljan Hang Xu Weiguo Gao Chunjing Xu Thomas B. Schon Li Zhang 18 161 0 22 Oct 2021
The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization Róbert Csordás Kazuki Irie Jürgen Schmidhuber AI4CE 33 55 0 14 Oct 2021