Anisotropy Is Inherent to Self-Attention in Transformers

22 January 2024

Papers citing "Anisotropy Is Inherent to Self-Attention in Transformers"

8 / 8 papers shown

Title
Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representations Yize Zhao Tina Behnia V. Vakilian Christos Thrampoulidis 55 8 0 20 Feb 2025
Discrete Speech Unit Extraction via Independent Component Analysis Tomohiko Nakamura Kwanghee Choi Keigo Hojo Yoshiaki Bando Satoru Fukayama Shinji Watanabe 43 0 0 11 Jan 2025
Norm of Mean Contextualized Embeddings Determines their Variance Hiroaki Yamagiwa Hidetoshi Shimodaira 27 0 0 17 Sep 2024
Transformer Layers as Painters Qi Sun Marc Pickett Aakash Kumar Nain Llion Jones AI4CE 31 13 0 12 Jul 2024
Outliers Dimensions that Disrupt Transformers Are Driven by Frequency Giovanni Puccetti Anna Rogers Aleksandr Drozd F. Dell’Orletta 76 42 0 23 May 2022
Low Anisotropy Sense Retrofitting (LASeR) : Towards Isotropic and Sense Enriched Representations Geetanjali Bihani Julia Taylor Rayz 31 13 0 22 Apr 2021
CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters Hicham El Boukkouri Olivier Ferret Thomas Lavergne Hiroshi Noji Pierre Zweigenbaum Junichi Tsujii 77 156 0 20 Oct 2020
ImageNet Large Scale Visual Recognition Challenge Olga Russakovsky Jia Deng Hao Su J. Krause S. Satheesh ... A. Karpathy A. Khosla Michael S. Bernstein Alexander C. Berg Li Fei-Fei VLM ObjD 296 39,198 0 01 Sep 2014