SHAPE: Shifted Absolute Position Embedding for Transformers

13 September 2021

Papers citing "SHAPE: Shifted Absolute Position Embedding for Transformers"

25 / 25 papers shown

Title
Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More More Arvid Frydenlund LRM 179 0 0 13 Mar 2025
Context-aware Biases for Length Extrapolation Ali Veisi Hamidreza Amirzadeh Amir Mansourian 165 1 0 11 Mar 2025
Learning the RoPEs: Better 2D and 3D Position Encodings with STRING Connor Schenck Isaac Reid M. Jacob Alex Bewley Joshua Ainslie ... Matthias Minderer Dmitry Kalashnikov Jonathan Tompson Vikas Sindhwani Krzysztof Choromanski 100 1 0 04 Feb 2025
Investigating Length Issues in Document-level Machine Translation Ziqian Peng Rachel Bawden François Yvon 108 2 0 23 Dec 2024
Rethinking Associative Memory Mechanism in Induction Head Shuo Wang Issei Sato 185 0 0 16 Dec 2024
HoPE: A Novel Positional Encoding Without Long-Term Decay for Enhanced Context Awareness and Extrapolation Yuhan Chen Ang Lv Jian Luan Bin Wang Wen Liu 66 5 0 28 Oct 2024
DAPE V2: Process Attention Score as Feature Map for Length Extrapolation Chuanyang Zheng Yihang Gao Han Shi Jing Xiong Jiankai Sun ... Xiaozhe Ren Michael Ng Xin Jiang Zhenguo Li Yu Li 83 3 0 07 Oct 2024
Language Models Need Inductive Biases to Count Inductively Yingshan Chang Yonatan Bisk LRM 78 7 0 30 May 2024
Base of RoPE Bounds Context Length Xin Men Mingyu Xu Bingning Wang Qingyu Zhang Hongyu Lin Xianpei Han Weipeng Chen 101 26 0 23 May 2024
Positional Encoding Helps Recurrent Neural Networks Handle a Large Vocabulary Takashi Morita 94 3 0 31 Jan 2024
The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Survey Saurav Pawar S.M. Towhidul Islam Tonmoy S. M. M. Zaman Vinija Jain Aman Chadha Amitava Das 68 29 0 15 Jan 2024
Positional Description Matters for Transformers Arithmetic Ruoqi Shen Sébastien Bubeck Ronen Eldan Yin Tat Lee Yuanzhi Li Yi Zhang 88 45 0 22 Nov 2023
Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey Yunpeng Huang Jingwei Xu Junyu Lai Zixu Jiang Taolue Chen ... Xiaoxing Ma Lijuan Yang Zhou Xin Shupeng Li Penghao Zhao LLMAG KELM 100 66 0 21 Nov 2023
Long-MIL: Scaling Long Contextual Multiple Instance Learning for Histopathology Whole Slide Image Analysis Honglin Li Yunlong Zhang Chenglu Zhu Jiatong Cai Sunyi Zheng Lin Yang VLM 89 4 0 21 Nov 2023
Extending Input Contexts of Language Models through Training on Segmented Sequences Petros Karypis Julian McAuley George Karypis 71 0 0 23 Oct 2023
LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models Chi Han Qifan Wang Hao Peng Wenhan Xiong Yu Chen Heng Ji Sinong Wang 157 61 0 30 Aug 2023
Length Generalization in Arithmetic Transformers Samy Jelassi Stéphane dÁscoli Carles Domingo-Enrich Yuhuai Wu Yuan-Fang Li Franccois Charton 100 43 0 27 Jun 2023
The Impact of Positional Encoding on Length Generalization in Transformers Amirhossein Kazemnejad Inkit Padhi Karthikeyan N. Ramamurthy Payel Das Siva Reddy 79 207 0 31 May 2023
Learning to Reason and Memorize with Self-Notes Jack Lanchantin Shubham Toshniwal Jason Weston Arthur Szlam Sainbayar Sukhbaatar ReLM LRM LLMAG 149 30 0 01 May 2023
The Curious Case of Absolute Position Embeddings Koustuv Sinha Amirhossein Kazemnejad Siva Reddy J. Pineau Dieuwke Hupkes Adina Williams 135 15 0 23 Oct 2022
Masked Jigsaw Puzzle: A Versatile Position Embedding for Vision Transformers Bin Ren Yahui Liu Yue Song Wei Bi Rita Cucchiara N. Sebe Wei Wang 122 28 0 25 May 2022
KERPLE: Kernelized Relative Positional Embedding for Length Extrapolation Ta-Chung Chi Ting-Han Fan Peter J. Ramadge Alexander I. Rudnicky 98 73 0 20 May 2022
Hilbert Flattening: a Locality-Preserving Matrix Unfolding Method for Visual Discrimination Qingsong Zhao Shuguang Dou Zhipeng Zhou Yangguang Li Yin Wang Yu Qiao Cairong Zhao 43 3 0 21 Feb 2022
Continual Transformers: Redundancy-Free Attention for Online Inference Lukas Hedegaard Arian Bakhtiarnia Alexandros Iosifidis CLL 98 12 0 17 Jan 2022
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press Noah A. Smith M. Lewis 350 779 0 27 Aug 2021