Relative Positional Encoding for Transformers with Linear Complexity

18 May 2021

Papers citing "Relative Positional Encoding for Transformers with Linear Complexity"

35 / 35 papers shown

Title
Of All StrIPEs: Investigating Structure-informed Positional Encoding for Efficient Music Generation Manvi Agarwal Changhong Wang Gaël Richard 29 0 0 07 Apr 2025
Positional Encoding in Transformer-Based Time Series Models: A Survey Habib Irani Vangelis Metsis AI4TS 53 0 0 17 Feb 2025
Linear Transformer Topological Masking with Graph Random Features Isaac Reid Kumar Avinava Dubey Deepali Jain Will Whitney Amr Ahmed ... Connor Schenck Richard E. Turner René Wagner Adrian Weller Krzysztof Choromanski 27 1 0 04 Oct 2024
ActionFlow: Equivariant, Accurate, and Efficient Policies with Spatially Symmetric Flow Matching Niklas Funk Julen Urain João Carvalho V. Prasad Georgia Chalvatzaki Jan Peters 50 5 0 06 Sep 2024
LeaPformer: Enabling Linear Transformers for Autoregressive and Simultaneous Tasks via Learned Proportions Victor Agostinelli Sanghyun Hong Lizhong Chen KELM 43 1 0 18 May 2024
Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: a Survey Dinh-Viet-Toan Le Louis Bigo Mikaela Keller Dorien Herremans MedIm 32 9 0 27 Feb 2024
Structure-informed Positional Encoding for Music Generation Manvi Agarwal Changhong Wang Gaël Richard 36 2 0 20 Feb 2024
Transformers as Graph-to-Graph Models James Henderson Alireza Mohammadshahi Andrei Catalin Coman Lesly Miculicich GNN 27 6 0 27 Oct 2023
Stochastic positional embeddings improve masked image modeling Amir Bar Florian Bordes Assaf Shocher Mahmoud Assran Pascal Vincent Nicolas Ballas Trevor Darrell Amir Globerson Yann LeCun 31 3 0 31 Jul 2023
Exploring Transformer Extrapolation Zhen Qin Yiran Zhong Huiyuan Deng 31 9 0 19 Jul 2023
Linearized Relative Positional Encoding Zhen Qin Weixuan Sun Kaiyue Lu Huizhong Deng Dong Li Xiaodong Han Yuchao Dai Lingpeng Kong Yiran Zhong 20 12 0 18 Jul 2023
Deep Multiple Instance Learning with Distance-Aware Self-Attention Georg Wolflein Lucie Charlotte Magister Pietro Lio' David J. Harrison Ognjen Arandjelovic 21 2 0 17 May 2023
Enhancing Multivariate Time Series Classifiers through Self-Attention and Relative Positioning Infusion Mehryar Abbasi Parvaneh Saeedi AI4TS 24 6 0 13 Feb 2023
Efficient Attention via Control Variates Lin Zheng Jianbo Yuan Chong-Jun Wang Lingpeng Kong 34 18 0 09 Feb 2023
Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers K. Choromanski Shanda Li Valerii Likhosherstov Kumar Avinava Dubey Shengjie Luo Di He Yiming Yang Tamás Sarlós Thomas Weingarten Adrian Weller 37 8 0 03 Feb 2023
Simplex Random Features Isaac Reid K. Choromanski Valerii Likhosherstov Adrian Weller 31 7 0 31 Jan 2023
Lightweight Structure-Aware Attention for Visual Understanding Heeseung Kwon F. M. Castro M. Marín-Jiménez N. Guil Alahari Karteek 28 2 0 29 Nov 2022
Melody Infilling with User-Provided Structural Context Chih-Pin Tan A. Su Yi-Hsuan Yang 36 3 0 06 Oct 2022
A Text Attention Network for Spatial Deformation Robust Scene Text Image Super-resolution Jianqi Ma Zhetong Liang Lei Zhang 25 43 0 17 Mar 2022
FastRPB: a Scalable Relative Positional Encoding for Long Sequence Tasks Maksim Zubkov Daniil Gavrilov 24 0 0 23 Feb 2022
cosFormer: Rethinking Softmax in Attention Zhen Qin Weixuan Sun Huicai Deng Dongxu Li Yunshen Wei Baohong Lv Junjie Yan Lingpeng Kong Yiran Zhong 24 212 0 17 Feb 2022
Finding Strong Gravitational Lenses Through Self-Attention H. Thuruthipilly A. Zadrożny Agnieszka Pollo Marek Biesiada 21 6 0 18 Oct 2021
Ripple Attention for Visual Perception with Sub-quadratic Complexity Lin Zheng Huijie Pan Lingpeng Kong 26 3 0 06 Oct 2021
Variable-Length Music Score Infilling via XLNet and Musically Specialized Positional Encoding Chin-Jui Chang Chun-Yi Lee Yi-Hsuan Yang 26 21 0 11 Aug 2021
From block-Toeplitz matrices to differential equations on graphs: towards a general theory for scalable masked Transformers K. Choromanski Han Lin Haoxian Chen Tianyi Zhang Arijit Sehanobish Valerii Likhosherstov Jack Parker-Holder Tamás Sarlós Adrian Weller Thomas Weingarten 29 33 0 16 Jul 2021
BERT-like Pre-training for Symbolic Piano Music Classification Tasks Yi-Hui Chou I-Chun Chen Chin-Jui Chang Joann Ching Yi-Hsuan Yang 32 25 0 12 Jul 2021
Learnable Fourier Features for Multi-Dimensional Spatial Positional Encoding Yang Li Si Si Gang Li Cho-Jui Hsieh Samy Bengio 16 86 0 05 Jun 2021
MuseMorphose: Full-Song and Fine-Grained Piano Music Style Transfer with One Transformer VAE Shih-Lun Wu Yi-Hsuan Yang ViT 25 53 0 10 May 2021
RoFormer: Enhanced Transformer with Rotary Position Embedding Jianlin Su Yu Lu Shengfeng Pan Ahmed Murtadha Bo Wen Yunfeng Liu 38 2,176 0 20 Apr 2021
SAPE: Spatially-Adaptive Progressive Encoding for Neural Optimization Amir Hertz Or Perel Raja Giryes O. Sorkine-Hornung Daniel Cohen-Or 28 67 0 19 Apr 2021
Position Information in Transformers: An Overview Philipp Dufter Martin Schmitt Hinrich Schütze 13 139 0 22 Feb 2021
LambdaNetworks: Modeling Long-Range Interactions Without Attention Irwan Bello 272 179 0 17 Feb 2021
Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs Wen-Yi Hsiao Jen-Yu Liu Yin-Cheng Yeh Yi-Hsuan Yang 113 180 0 07 Jan 2021
Efficient Content-Based Sparse Attention with Routing Transformers Aurko Roy M. Saffar Ashish Vaswani David Grangier MoE 246 580 0 12 Mar 2020
DDSP: Differentiable Digital Signal Processing Jesse Engel Lamtharn Hantrakul Chenjie Gu Adam Roberts DiffM 94 373 0 14 Jan 2020