IsoScore: Measuring the Uniformity of Embedding Space Utilization

v1v2 (latest)

IsoScore: Measuring the Uniformity of Embedding Space Utilization

16 August 2021

Carsten Eickhoff

ArXiv (abs)PDF HTML

Papers citing "IsoScore: Measuring the Uniformity of Embedding Space Utilization"

14 / 14 papers shown

Title
Dissecting Query-Key Interaction in Vision Transformers Xu Pan Aaron Philip Ziqian Xie Odelia Schwartz 99 1 0 04 Apr 2024
Learning to Remove: Towards Isotropic Pre-trained BERT Embedding Y. Liang Rui Cao Jie Zheng Jie Ren Ling Gao SSL 142 28 0 12 Apr 2021
IsoBN: Fine-Tuning BERT with Isotropic Batch Normalization Wenxuan Zhou Bill Yuchen Lin Xiang Ren 75 25 0 02 May 2020
What do you mean, BERT? Assessing BERT as a Distributional Semantics Model Timothee Mickus Denis Paperno Mathieu Constant Kees van Deemter 58 46 0 13 Nov 2019
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter Victor Sanh Lysandre Debut Julien Chaumond Thomas Wolf 234 7,520 0 02 Oct 2019
How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings Kawin Ethayarajh 86 872 0 02 Sep 2019
Representation Degeneration Problem in Training Natural Language Generation Models Jun Gao Di He Xu Tan Tao Qin Liwei Wang Tie-Yan Liu 62 268 0 28 Jul 2019
Improving Neural Language Modeling via Adversarial Training Dilin Wang Chengyue Gong Qiang Liu AAML 81 118 0 10 Jun 2019
Visualizing and Measuring the Geometry of BERT Andy Coenen Emily Reif Ann Yuan Been Kim Adam Pearce F. Viégas Martin Wattenberg MILM 78 417 0 06 Jun 2019
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 1.8K 94,891 0 11 Oct 2018
FRAGE: Frequency-Agnostic Word Representation Chengyue Gong Di He Xu Tan Tao Qin Liwei Wang Tie-Yan Liu OOD 64 144 0 18 Sep 2018
All-but-the-Top: Simple and Effective Postprocessing for Word Representations Jiaqi Mu S. Bhat Pramod Viswanath 70 311 0 05 Feb 2017
Pointer Sentinel Mixture Models Stephen Merity Caiming Xiong James Bradbury R. Socher RALM 328 2,876 0 26 Sep 2016
A Latent Variable Model Approach to PMI-based Word Embeddings Sanjeev Arora Yuanzhi Li Yingyu Liang Tengyu Ma Andrej Risteski 48 58 0 12 Feb 2015