How not to Lie with a Benchmark: Rearranging NLP Leaderboards

How not to Lie with a Benchmark: Rearranging NLP Leaderboards

2 December 2021

Tatiana Shavrina

Valentin Malykh

ArXiv (abs)PDF HTML

Papers citing "How not to Lie with a Benchmark: Rearranging NLP Leaderboards"

13 / 13 papers shown

Title
The Benchmark Lottery Mostafa Dehghani Yi Tay A. Gritsenko Zhe Zhao N. Houlsby Fernando Diaz Donald Metzler Oriol Vinyals 96 92 0 14 Jul 2021
Memorization vs. Generalization: Quantifying Data Leakage in NLP Performance Evaluation Aparna Elangovan Jiayuan He Karin Verspoor TDI FedML 204 95 0 03 Feb 2021
Utility is in the Eye of the User: A Critique of NLP Leaderboards Kawin Ethayarajh Dan Jurafsky ELM 77 52 0 29 Sep 2020
DeBERTa: Decoding-enhanced BERT with Disentangled Attention Pengcheng He Xiaodong Liu Jianfeng Gao Weizhu Chen AAML 161 2,747 0 05 Jun 2020
XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation Yaobo Liang Nan Duan Yeyun Gong Ning Wu Fenfei Guo ... Shuguang Liu Fan Yang Daniel Fernando Campos Rangan Majumder Ming Zhou ELM VLM 86 350 0 03 Apr 2020
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization Junjie Hu Sebastian Ruder Aditya Siddhant Graham Neubig Orhan Firat Melvin Johnson ELM 183 975 0 24 Mar 2020
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 447 20,298 0 23 Oct 2019
Show Your Work: Improved Reporting of Experimental Results Jesse Dodge Suchin Gururangan Dallas Card Roy Schwartz Noah A. Smith 72 255 0 06 Sep 2019
Human vs. Muppet: A Conservative Estimate of Human Performance on the GLUE Benchmark Nikita Nangia Samuel R. Bowman ELM ALM 69 76 0 24 May 2019
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions Christopher Clark Kenton Lee Ming-Wei Chang Tom Kwiatkowski Michael Collins Kristina Toutanova 227 1,549 0 24 May 2019
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems Alex Jinpeng Wang Yada Pruksachatkun Nikita Nangia Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 274 2,315 0 02 May 2019
The Natural Language Decathlon: Multitask Learning as Question Answering Bryan McCann N. Keskar Caiming Xiong R. Socher AIMat MLLM BDL 147 645 0 20 Jun 2018
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 1.1K 7,182 0 20 Apr 2018

We use cookies and other tracking technologies to improve your browsing experience on our website, to show you personalized content and targeted ads, to analyze our website traffic, and to understand where our visitors are coming from. See our policy.