Preference Leakage: A Contamination Problem in LLM-as-a-judge

3 February 2025

Papers citing "Preference Leakage: A Contamination Problem in LLM-as-a-judge"

37 / 87 papers shown

Title
LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models Yaowei Zheng Richong Zhang Junhao Zhang Yanhan Ye Zheyan Luo Zhangchi Feng Yongqiang Ma 139 515 0 20 Mar 2024
Elephants Never Forget: Testing Language Models for Memorization of Tabular Data Sebastian Bordt Harsha Nori Rich Caruana LMTD 69 18 0 11 Mar 2024
Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models Yihong Dong Xue Jiang Huanyu Liu Zhi Jin Bin Gu Mengfei Yang Ge Li 57 54 0 24 Feb 2024
Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinement Wenda Xu Guanglei Zhu Xuandong Zhao Liangming Pan Lei Li Wenjie Wang 62 60 0 18 Feb 2024
Humans or LLMs as the Judge? A Study on Judgement Biases Guiming Hardy Chen Shunian Chen Ziche Liu Feng Jiang Benyou Wang 103 107 0 16 Feb 2024
Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning Ming Li Lichang Chen Jiuhai Chen Shwai He Jiuxiang Gu Dinesh Manocha 113 55 0 15 Feb 2024
Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluation Xiaoying Zhang Baolin Peng Ye Tian Jingyan Zhou Lifeng Jin Linfeng Song Haitao Mi Helen Meng HILM 67 51 0 14 Feb 2024
Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs Simone Balloccu Patrícia Schmidtová Mateusz Lango Ondrej Dusek SILM ELM PILM 69 174 0 06 Feb 2024
Contextualization Distillation from Large Language Model for Knowledge Graph Completion Dawei Li Zhen Tan Tianlong Chen Huan Liu KELM 63 13 0 28 Jan 2024
Investigating Data Contamination for Pre-training Language Models Minhao Jiang Ken Ziyu Liu Ming Zhong Rylan Schaeffer Siru Ouyang Jiawei Han Sanmi Koyejo 64 71 0 11 Jan 2024
LLM-as-a-Coauthor: Can Mixed Human-Written and Machine-Generated Text Be Detected? Qihui Zhang Chujie Gao Dongping Chen Yue Huang Yixin Huang ... Shilin Zhang Weiye Li Zhengyan Fu Yao Wan Lichao Sun DeLMO 30 31 0 11 Jan 2024
Task Contamination: Language Models May Not Be Few-Shot Anymore Changmao Li Jeffrey Flanigan 143 101 0 26 Dec 2023
What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning Wei Liu Weihao Zeng Keqing He Yong Jiang Junxian He ALM 94 231 0 25 Dec 2023
AlignBench: Benchmarking Chinese Alignment of Large Language Models Xiao Liu Xuanyu Lei Sheng-Ping Wang Yue Huang Zhuoer Feng ... Hongning Wang Jing Zhang Minlie Huang Yuxiao Dong Jie Tang ELM LM&MA ALM 143 49 0 30 Nov 2023
Investigating Data Contamination in Modern Benchmarks for Large Language Models Chunyuan Deng Yilun Zhao Xiangru Tang Mark B. Gerstein Arman Cohan AAML ELM 54 59 0 16 Nov 2023
LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scores Yiqi Liu N. Moosavi Chenghua Lin ELM 70 53 0 16 Nov 2023
Ziya2: Data-centric Learning is All LLMs Need Ruyi Gan Ziwei Wu Renliang Sun Junyu Lu Xiaojun Wu ... Ping Yang Qi Yang Hao Wang Jiaxing Zhang Yan Song VLM ALM 70 19 0 06 Nov 2023
Prometheus: Inducing Fine-grained Evaluation Capability in Language Models Seungone Kim Jamin Shin Yejin Cho Joel Jang Shayne Longpre ... Sangdoo Yun Seongjin Shin Sungdong Kim James Thorne Minjoon Seo ALM LM&MA ELM 67 236 0 12 Oct 2023
Mistral 7B Albert Q. Jiang Alexandre Sablayrolles A. Mensch Chris Bamford Devendra Singh Chaplot ... Teven Le Scao Thibaut Lavril Thomas Wang Timothée Lacroix William El Sayed MoE LRM 74 2,197 0 10 Oct 2023
MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use Yue Huang Jiawen Shi Yuan Li Chenrui Fan Siyuan Wu ... Yixin Liu Pan Zhou Yao Wan Neil Zhenqiang Gong Lichao Sun LLMAG 59 91 0 04 Oct 2023
Benchmarking Cognitive Biases in Large Language Models as Evaluators Ryan Koo Minhwa Lee Vipul Raheja Jong Inn Park Zae Myung Kim Dongyeop Kang ALM 66 83 0 29 Sep 2023
Time Travel in LLMs: Tracing Data Contamination in Large Language Models Shahriar Golchin Mihai Surdeanu 103 102 0 16 Aug 2023
AgentBench: Evaluating LLMs as Agents Xiao Liu Hao Yu Hanchen Zhang Yifan Xu Xuanyu Lei ... Yu-Chuan Su Huan Sun Minlie Huang Yuxiao Dong Jie Tang ELM LLMAG 127 297 0 07 Aug 2023
Won't Get Fooled Again: Answering Questions with False Premises Shengding Hu Yi-Xiao Luo Huadong Wang Xingyi Cheng Zhiyuan Liu Maosong Sun 57 28 0 05 Jul 2023
Assisting Language Learners: Automated Trans-Lingual Definition Generation via Contrastive Prompt Learning Hengyuan Zhang Dawei Li Yanran Li Chenming Shang Chufan Shi Yong Jiang 126 15 0 09 Jun 2023
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena Lianmin Zheng Wei-Lin Chiang Ying Sheng Siyuan Zhuang Zhanghao Wu ... Dacheng Li Eric Xing Haotong Zhang Joseph E. Gonzalez Ion Stoica ALM OSLM ELM 327 4,298 0 09 Jun 2023
A New Dataset and Empirical Study for Sentence Simplification in Chinese Shiping Yang Renliang Sun Xiao-Yi Wan 50 9 0 07 Jun 2023
Direct Preference Optimization: Your Language Model is Secretly a Reward Model Rafael Rafailov Archit Sharma E. Mitchell Stefano Ermon Christopher D. Manning Chelsea Finn ALM 379 3,942 0 29 May 2023
OpenAssistant Conversations -- Democratizing Large Language Model Alignment Andreas Kopf Yannic Kilcher Dimitri von Rutte Sotiris Anagnostidis Zhi Rui Tam ... Arnav Dantuluri Andrew Maguire Christoph Schuhmann Huu Nguyen A. Mattick ALM LM&MA 126 628 0 14 Apr 2023
Human-like Summarization Evaluation with ChatGPT Mingqi Gao Jie Ruan Renliang Sun Xunjian Yin Shiping Yang Xiaojun Wan ALM AI4MH 55 132 0 05 Apr 2023
GPT-4 Technical Report OpenAI OpenAI OpenAI Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad ... Shengjia Zhao Tianhao Zheng Juntang Zhuang William Zhuk Barret Zoph LLMAG MLLM 1.4K 14,313 0 15 Mar 2023
Towards a Unified Multi-Dimensional Evaluator for Text Generation Ming Zhong Yang Liu Da Yin Yuning Mao Yizhu Jiao Peng Liu Chenguang Zhu Heng Ji Jiawei Han ELM 67 272 0 13 Oct 2022
TruthfulQA: Measuring How Models Mimic Human Falsehoods Stephanie C. Lin Jacob Hilton Owain Evans HILM 135 1,897 0 08 Sep 2021
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus Jesse Dodge Maarten Sap Ana Marasović William Agnew Gabriel Ilharco Dirk Groeneveld Margaret Mitchell Matt Gardner AILaw 112 446 0 18 Apr 2021
Memorization vs. Generalization: Quantifying Data Leakage in NLP Performance Evaluation Aparna Elangovan Jiayuan He Karin Verspoor TDI FedML 201 93 0 03 Feb 2021
BERTScore: Evaluating Text Generation with BERT Tianyi Zhang Varsha Kishore Felix Wu Kilian Q. Weinberger Yoav Artzi 312 5,801 0 21 Apr 2019
How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation Chia-Wei Liu Ryan J. Lowe Iulian Serban Michael Noseworthy Laurent Charlin Joelle Pineau 104 1,295 0 25 Mar 2016