What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding

10 October 2020

Papers citing "What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding"

18 / 18 papers shown

Title
AttentionSmithy: A Modular Framework for Rapid Transformer Development and Customization Caleb Cranney Jesse G. Meyer 85 0 0 13 Feb 2025
TEE4EHR: Transformer Event Encoder for Better Representation Learning in Electronic Health Records Hojjat Karami David Atienza Anisoara Ionescu AI4TS 38 1 0 09 Feb 2024
Investigating the Effect of Relative Positional Embeddings on AMR-to-Text Generation with Structural Adapters Sébastien Montella Alexis Nasr Johannes Heinecke Frédéric Béchet L. Rojas-Barahona 29 2 0 12 Feb 2023
Position Embedding Needs an Independent Layer Normalization Runyi Yu Zhennan Wang Yinhuai Wang Kehan Li Yian Zhao Jian Zhang Guoli Song Jie Chen 31 1 0 10 Dec 2022
Rethinking Video ViTs: Sparse Video Tubes for Joint Image and Video Learning A. Piergiovanni Weicheng Kuo A. Angelova ViT 38 54 0 06 Dec 2022
The Curious Case of Absolute Position Embeddings Koustuv Sinha Amirhossein Kazemnejad Siva Reddy J. Pineau Dieuwke Hupkes Adina Williams 87 15 0 23 Oct 2022
Experiencer-Specific Emotion and Appraisal Prediction Maximilian Wegge Enrica Troiano Laura Oberländer Roman Klinger 37 7 0 21 Oct 2022
Deep Transformer Q-Networks for Partially Observable Reinforcement Learning Kevin Esslinger Robert W. Platt Chris Amato OffRL 35 35 0 02 Jun 2022
ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts Akari Asai Mohammadreza Salehi Matthew E. Peters Hannaneh Hajishirzi 130 100 0 24 May 2022
UniTE: Unified Translation Evaluation Boyi Deng Dayiheng Liu Baosong Yang Haibo Zhang Boxing Chen Derek F. Wong Lidia S. Chao 41 41 0 28 Apr 2022
Topic Detection and Tracking with Time-Aware Document Embeddings Hang Jiang Doug Beeferman Weiquan Mao D. Roy AI4TS 21 1 0 12 Dec 2021
Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbations Ekaterina Taktasheva Vladislav Mikhailov Ekaterina Artemova 24 13 0 28 Sep 2021
RuleBert: Teaching Soft Rules to Pre-trained Language Models Mohammed Saeed N. Ahmadi Preslav Nakov Paolo Papotti LRM 261 32 0 24 Sep 2021
The Grammar-Learning Trajectories of Neural Language Models Leshem Choshen Guy Hacohen D. Weinshall Omri Abend 31 28 0 13 Sep 2021
The Case for Translation-Invariant Self-Attention in Transformer-Based Language Models Ulme Wennberg G. Henter MILM 35 21 0 03 Jun 2021
Relative Positional Encoding for Transformers with Linear Complexity Antoine Liutkus Ondřej Cífka Shih-Lun Wu Umut Simsekli Yi-Hsuan Yang Gaël Richard 38 45 0 18 May 2021
"Average" Approximates "First Principal Component"? An Empirical Analysis on Representations from Neural Language Models Zihan Wang Chengyu Dong Jingbo Shang FAtt 42 4 0 18 Apr 2021
Position Information in Transformers: An Overview Philipp Dufter Martin Schmitt Hinrich Schütze 18 141 0 22 Feb 2021