NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark

27 October 2023

Oscar Sainz

Jon Ander Campos

Iker García-Ferrero

Julen Etxaniz

Oier López de Lacalle

Eneko Agirre

ArXiv PDF HTML

Papers citing "NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark"

43 / 43 papers shown

Title
Towards Contamination Resistant Benchmarks Rahmatullah Musawi Sheng Lu 42 0 0 13 May 2025
Position: AI Competitions Provide the Gold Standard for Empirical Rigor in GenAI Evaluation D. Sculley Will Cukierski Phil Culliton Sohier Dane Maggie Demkin ... Addison Howard Paul Mooney Walter Reade Megan Risdal Nate Keating 31 0 0 01 May 2025
Large Language Models Could Be Rote Learners Yuyang Xu Renjun Hu Haochao Ying Jian Wu Xing Shi Wei Lin ELM 166 0 0 11 Apr 2025
A Perplexity and Menger Curvature-Based Approach for Similarity Evaluation of Large Language Models Yuantao Zhang Zhankui Yang AAML 38 0 0 05 Apr 2025
Say Less, Mean More: Leveraging Pragmatics in Retrieval-Augmented Generation Haris Riaz Ellen Riloff Mihai Surdeanu RALM 51 0 0 25 Feb 2025
None of the Others: a General Technique to Distinguish Reasoning from Memorization in Multiple-Choice LLM Evaluation Benchmarks Eva Sánchez Salido Julio Gonzalo Guillermo Marco ELM 60 2 0 18 Feb 2025
MathGAP: Out-of-Distribution Evaluation on Problems with Arbitrarily Complex Proofs Andreas Opedal Haruki Shirakami Bernhard Schölkopf Abulhair Saparov Mrinmaya Sachan LRM 57 1 0 17 Feb 2025
Unbiased Evaluation of Large Language Models from a Causal Perspective Meilin Chen Jian Tian Liang Ma Di Xie Weijie Chen Jiang Zhu ALM ELM 54 0 0 10 Feb 2025
Self-Rationalization in the Wild: A Large Scale Out-of-Distribution Evaluation on NLI-related tasks Jing Yang Max Glockner Anderson de Rezende Rocha Iryna Gurevych LRM 73 1 0 07 Feb 2025
Benchmarking LLMs' Judgments with No Gold Standard Shengwei Xu Yuxuan Lu Grant Schoenebeck Yuqing Kong 34 1 0 11 Nov 2024
Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptions Yujuan Fu Özlem Uzuner Meliha Yetisgen Fei Xia 59 3 0 24 Oct 2024
Detecting Training Data of Large Language Models via Expectation Maximization Gyuwan Kim Yang Li Evangelia Spiliopoulou Jie Ma Miguel Ballesteros William Yang Wang MIALM 95 4 2 10 Oct 2024
Fine-tuning can Help Detect Pretraining Data from Large Language Models H. Zhang Songxin Zhang Bingyi Jing Hongxin Wei 43 0 0 09 Oct 2024
ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities Ezra Karger Houtan Bastani Chen Yueh-Han Zachary Jacobs Danny Halawi Fred Zhang P. Tetlock 35 6 0 30 Sep 2024
Lessons Learned on Information Retrieval in Electronic Health Records: A Comparison of Embedding Models and Pooling Strategies Skatje Myers Timothy A. Miller Yanjun Gao M. Churpek Anoop Mayampurath Dmitriy Dligach Majid Afshar 28 3 0 23 Sep 2024
Bilingual Evaluation of Language Models on General Knowledge in University Entrance Exams with Minimal Contamination Eva Sánchez Salido Roser Morante Julio Gonzalo Guillermo Marco Jorge Carrillo-de-Albornoz ... Enrique Amigó Andrés Fernández Alejandro Benito-Santos Adrián Ghajari Espinosa Victor Fresno ELM 51 0 0 19 Sep 2024
ASR Error Correction using Large Language Models Rao Ma Mengjie Qian Mark J. F. Gales Kate Knill KELM 46 1 0 14 Sep 2024
Con-ReCall: Detecting Pre-training Data in LLMs via Contrastive Decoding Cheng Wang Yiwei Wang Bryan Hooi Yujun Cai Nanyun Peng Kai-Wei Chang 42 2 0 05 Sep 2024
Training on the Test Task Confounds Evaluation and Emergence Ricardo Dominguez-Olmedo Florian E. Dorner Moritz Hardt ELM 71 7 1 10 Jul 2024
A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations Md Tahmid Rahman Laskar Sawsan Alqahtani M Saiful Bari Mizanur Rahman Mohammad Abdullah Matin Khan ... Chee Wei Tan Md. Rizwan Parvez Enamul Hoque Shafiq R. Joty Jimmy Huang ELM ALM 29 28 0 04 Jul 2024
PaCoST: Paired Confidence Significance Testing for Benchmark Contamination Detection in Large Language Models Huixuan Zhang Yun Lin Xiaojun Wan 50 0 0 26 Jun 2024
VarBench: Robust Language Model Benchmarking Through Dynamic Variable Perturbation Kun Qian Shunji Wan Claudia Tang Youzhi Wang Xuanming Zhang Maximillian Chen Zhou Yu AAML 45 8 0 25 Jun 2024
The Unlikely Duel: Evaluating Creative Writing in LLMs through a Unique Scenario Carlos Gómez-Rodríguez Paul Williams 27 1 0 22 Jun 2024
BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack Yuri Kuratov Aydar Bulatov Petr Anokhin Ivan Rodkin Dmitry Sorokin Artyom Sorokin Andrey Kravchenko RALM ALM LRM ReLM ELM 49 59 0 14 Jun 2024
Benchmark Data Contamination of Large Language Models: A Survey Cheng Xu Shuhao Guan Derek Greene Mohand-Tahar Kechadi ELM ALM 38 39 0 06 Jun 2024
Large Language Models are Zero-Shot Next Location Predictors Ciro Beneduce Bruno Lepri Massimiliano Luca 41 7 0 31 May 2024
Eliciting Informative Text Evaluations with Large Language Models Yuxuan Lu Shengwei Xu Yichi Zhang Yuqing Kong Grant Schoenebeck 39 5 0 23 May 2024
A Multi-Perspective Analysis of Memorization in Large Language Models Bowen Chen Namgi Han Yusuke Miyao 46 1 0 19 May 2024
BMRetriever: Tuning Large Language Models as Better Biomedical Text Retrievers Ran Xu Wenqi Shi Yue Yu Yuchen Zhuang Yanqiao Zhu M. D. Wang Joyce C. Ho Chao Zhang Carl Yang LM&MA 40 19 0 29 Apr 2024
Hatred Stems from Ignorance! Distillation of the Persuasion Modes in Countering Conversational Hate Speech Ghadi Alyahya Abeer Aldayel 43 2 0 18 Mar 2024
Visual Hallucinations of Multi-modal Large Language Models Wen Huang Hongbin Liu Minxin Guo Neil Zhenqiang Gong MLLM VLM 32 24 0 22 Feb 2024
GeoEval: Benchmark for Evaluating LLMs and Multi-Modal Models on Geometry Problem-Solving Jiaxin Zhang Zhong-Zhi Li Ming-Liang Zhang Fei Yin Cheng-Lin Liu Yashar Moshfeghi ELM LRM 46 17 0 15 Feb 2024
PROXYQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Models Haochen Tan Zhijiang Guo Zhan Shi Lu Xu Zhili Liu ... Xiaoguang Li Yasheng Wang Lifeng Shang Qun Liu Linqi Song 38 12 0 26 Jan 2024
Does Pre-trained Language Model Actually Infer Unseen Links in Knowledge Graph Completion? Yusuke Sakai Hidetaka Kamigaito Katsuhiko Hayashi Taro Watanabe 26 1 0 15 Nov 2023
Pitfalls in Link Prediction with Graph Neural Networks: Understanding the Impact of Target-link Inclusion & Better Practices Jing Zhu Yuhang Zhou V. Ioannidis Sheng Qian Wei Ai Xiang Song Danai Koutra 25 9 0 01 Jun 2023
DPIC: Decoupling Prompt and Intrinsic Characteristics for LLM Generated Text Detection Xiao Yu Yuang Qi Kejiang Chen Guoqiang Chen Xi Yang Pengyuan Zhu Xiuwei Shang Weiming Zhang Neng H. Yu DeLMO 13 11 0 21 May 2023
Data Portraits: Recording Foundation Model Training Data Marc Marone Benjamin Van Durme 143 30 0 06 Mar 2023
PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts Stephen H. Bach Victor Sanh Zheng-Xin Yong Albert Webson Colin Raffel ... Khalid Almubarak Xiangru Tang Dragomir R. Radev Mike Tian-Jian Jiang Alexander M. Rush VLM 225 338 0 02 Feb 2022
Deduplicating Training Data Makes Language Models Better Katherine Lee Daphne Ippolito A. Nystrom Chiyuan Zhang Douglas Eck Chris Callison-Burch Nicholas Carlini SyDa 242 593 0 14 Jul 2021
Memorization vs. Generalization: Quantifying Data Leakage in NLP Performance Evaluation Aparna Elangovan Jiayuan He Karin Verspoor TDI FedML 167 89 0 03 Feb 2021
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 264 4,489 0 23 Jan 2020
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 297 6,959 0 20 Apr 2018
Teaching Machines to Read and Comprehend Karl Moritz Hermann Tomás Kociský Edward Grefenstette L. Espeholt W. Kay Mustafa Suleyman Phil Blunsom 184 3,510 0 10 Jun 2015