The Curious Case of Absolute Position Embeddings

23 October 2022

Koustuv Sinha

Amirhossein Kazemnejad

Siva Reddy

Papers citing "The Curious Case of Absolute Position Embeddings"

33 / 33 papers shown

Title
Conversation AI Dialog for Medicare powered by Finetuning and Retrieval Augmented Generation Atharva Mangeshkumar Agrawal Rutika Pandurang Shinde Vasanth Kumar Bhukya Ashmita Chakraborty Sagar Bharat Shah Tanmay Shukla Sree Pradeep Kumar Relangi Nilesh Mutyam LM&MA AI4MH 112 0 0 04 Feb 2025
OPT: Open Pre-trained Transformer Language Models Susan Zhang Stephen Roller Naman Goyal Mikel Artetxe Moya Chen ... Daniel Simig Punit Singh Koura Anjali Sridhar Tianlu Wang Luke Zettlemoyer VLM OSLM AI4CE 327 3,667 0 02 May 2022
GPT-NeoX-20B: An Open-Source Autoregressive Language Model Sid Black Stella Biderman Eric Hallahan Quentin G. Anthony Leo Gao ... Shivanshu Purohit Laria Reynolds J. Tow Benqi Wang Samuel Weinbach 171 829 0 14 Apr 2022
PaLM: Scaling Language Modeling with Pathways Aakanksha Chowdhery Sharan Narang Jacob Devlin Maarten Bosma Gaurav Mishra ... Kathy Meier-Hellstern Douglas Eck J. Dean Slav Petrov Noah Fiedel PILM LRM 480 6,240 0 05 Apr 2022
Transformer Language Models without Positional Encodings Still Learn Positional Information Adi Haviv Ori Ram Ofir Press Peter Izsak Omer Levy 94 124 0 30 Mar 2022
minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Models Kanishka Misra 58 62 0 24 Mar 2022
Efficient Large Scale Language Modeling with Mixtures of Experts Mikel Artetxe Shruti Bhosale Naman Goyal Todor Mihaylov Myle Ott ... Jeff Wang Luke Zettlemoyer Mona T. Diab Zornitsa Kozareva Ves Stoyanov MoE 183 195 0 20 Dec 2021
SHAPE: Shifted Absolute Position Embedding for Transformers Shun Kiyono Sosuke Kobayashi Jun Suzuki Kentaro Inui 261 46 0 13 Sep 2021
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press Noah A. Smith M. Lewis 324 759 0 27 Aug 2021
The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers Róbert Csordás Kazuki Irie Jürgen Schmidhuber ViT 68 133 0 26 Aug 2021
Do Vision Transformers See Like Convolutional Neural Networks? M. Raghu Thomas Unterthiner Simon Kornblith Chiyuan Zhang Alexey Dosovitskiy ViT 132 958 0 19 Aug 2021
Making Transformers Solve Compositional Tasks Santiago Ontañón Joshua Ainslie Vaclav Cvicek Zachary Kenneth Fisher 92 73 0 09 Aug 2021
Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little Koustuv Sinha Robin Jia Dieuwke Hupkes J. Pineau Adina Williams Douwe Kiela 83 246 0 14 Apr 2021
Shortformer: Better Language Modeling using Shorter Inputs Ofir Press Noah A. Smith M. Lewis 258 90 0 31 Dec 2020
Positional Artefacts Propagate Through Masked Language Model Embeddings Ziyang Luo Artur Kulmizev Xiaoxi Mao 75 41 0 09 Nov 2020
What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding Yu-An Wang Yun-Nung Chen SSL 46 95 0 10 Oct 2020
Exploring BERT's Sensitivity to Lexical Cues using Tests from Semantic Priming Kanishka Misra Allyson Ettinger Julia Taylor Rayz 44 58 0 06 Oct 2020
Rethinking Positional Encoding in Language Pre-training Guolin Ke Di He Tie-Yan Liu 69 295 0 28 Jun 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 762 42,055 0 28 May 2020
Longformer: The Long-Document Transformer Iz Beltagy Matthew E. Peters Arman Cohan RALM VLM 168 4,071 0 10 Apr 2020
BLiMP: The Benchmark of Linguistic Minimal Pairs for English Alex Warstadt Alicia Parrish Haokun Liu Anhad Mohananey Wei Peng Sheng-Fu Wang Samuel R. Bowman 75 491 0 02 Dec 2019
PIQA: Reasoning about Physical Commonsense in Natural Language Yonatan Bisk Rowan Zellers Ronan Le Bras Jianfeng Gao Yejin Choi OOD LRM 144 1,806 0 26 Nov 2019
Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly Nora Kassner Hinrich Schütze 68 323 0 08 Nov 2019
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension M. Lewis Yinhan Liu Naman Goyal Marjan Ghazvininejad Abdel-rahman Mohamed Omer Levy Veselin Stoyanov Luke Zettlemoyer AIMat VLM 249 10,829 0 29 Oct 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 424 20,181 0 23 Oct 2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy M. Lewis Luke Zettlemoyer Veselin Stoyanov AIMat 653 24,464 0 26 Jul 2019
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context Zihang Dai Zhilin Yang Yiming Yang J. Carbonell Quoc V. Le Ruslan Salakhutdinov VLM 241 3,728 0 09 Jan 2019
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 1.8K 94,891 0 11 Oct 2018
Neural Network Acceptability Judgments Alex Warstadt Amanpreet Singh Samuel R. Bowman 230 1,407 0 31 May 2018
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 1.1K 7,159 0 20 Apr 2018
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge Peter Clark Isaac Cowhey Oren Etzioni Tushar Khot Ashish Sabharwal Carissa Schoenick Oyvind Tafjord ELM RALM LRM 158 2,610 0 14 Mar 2018
Self-Attention with Relative Position Representations Peter Shaw Jakob Uszkoreit Ashish Vaswani 174 2,290 0 06 Mar 2018
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 701 131,652 0 12 Jun 2017