Assistant-Guided Mitigation of Teacher Preference Bias in LLM-as-a-Judge

v1v2 (latest)

Assistant-Guided Mitigation of Teacher Preference Bias in LLM-as-a-Judge

25 May 2025

ArXiv (abs)PDF HTML

Papers citing "Assistant-Guided Mitigation of Teacher Preference Bias in LLM-as-a-Judge"

19 / 19 papers shown

Title
Do LLM Evaluators Prefer Themselves for a Reason? Wei-Lin Chen Zhepei Wei Xinyu Zhu Shi Feng Yu Meng ELM LRM 89 3 0 04 Apr 2025
PRD: Peer Rank and Discussion Improve Large Language Model based Evaluations Ruosen Li Teerth Patel Xinya Du LLMAG ALM 185 102 0 03 Jan 2025
Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge Jiayi Ye Yanbo Wang Yue Huang Dongping Chen Qihui Zhang ... Werner Geyer Chao Huang Pin-Yu Chen Nitesh Chawla Xiangliang Zhang ELM 121 78 0 03 Oct 2024
Aligning Human and LLM Judgments: Insights from EvalAssist on Task-Specific Evaluations and AI-assisted Assessment Strategy Preferences Zahra Ashktorab Michael Desmond Qian Pan James M. Johnson Martin Santillan Cooper Elizabeth M. Daly Rahul Nair Tejaswini Pedapati Swapnaja Achintalwar Werner Geyer ELM 105 7 0 01 Oct 2024
Beyond Scalar Reward Model: Learning Generative Judge from Preference Data Ziyi Ye Xiangsheng Li Qiuchi Li Qingyao Ai Yujia Zhou Wei Shen Dong Yan Yiqun Liu 120 17 0 01 Oct 2024
Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge Tianhao Wu Weizhe Yuan O. Yu. Golovneva Jing Xu Yuandong Tian Jiantao Jiao Jason Weston Sainbayar Sukhbaatar ALM KELM LRM 142 96 0 28 Jul 2024
LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks A. Bavaresco Raffaella Bernardi Leonardo Bertolazzi Desmond Elliott Raquel Fernández ... David Schlangen Alessandro Suglia Aditya K Surikuchi Ece Takmaz A. Testoni ALM ELM 177 88 0 26 Jun 2024
Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models Seungone Kim Juyoung Suk Shayne Longpre Bill Yuchen Lin Jamin Shin Sean Welleck Graham Neubig Moontae Lee Kyungjae Lee Minjoon Seo MoMe ALM ELM 139 205 0 02 May 2024
LLM Evaluators Recognize and Favor Their Own Generations Arjun Panickssery Samuel R. Bowman Shi Feng 122 197 0 15 Apr 2024
Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators Yann Dubois Balázs Galambosi Percy Liang Tatsunori Hashimoto ALM 164 403 0 06 Apr 2024
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference Wei-Lin Chiang Lianmin Zheng Ying Sheng Anastasios Nikolas Angelopoulos Tianle Li ... Hao Zhang Banghua Zhu Michael I. Jordan Joseph E. Gonzalez Ion Stoica OSLM 160 602 0 07 Mar 2024
Humans or LLMs as the Judge? A Study on Judgement Biases Guiming Hardy Chen Shunian Chen Ziche Liu Feng Jiang Benyou Wang 206 113 0 16 Feb 2024
JudgeLM: Fine-tuned Large Language Models are Scalable Judges Lianghui Zhu Xinggang Wang Xinlong Wang ELM ALM 156 142 0 26 Oct 2023
Prometheus: Inducing Fine-grained Evaluation Capability in Language Models Seungone Kim Jamin Shin Yejin Cho Joel Jang Shayne Longpre ... Sangdoo Yun Seongjin Shin Sungdong Kim James Thorne Minjoon Seo ALM LM&MA ELM 110 240 0 12 Oct 2023
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena Lianmin Zheng Wei-Lin Chiang Ying Sheng Siyuan Zhuang Zhanghao Wu ... Dacheng Li Eric Xing Haotong Zhang Joseph E. Gonzalez Ion Stoica ALM OSLM ELM 531 4,453 0 09 Jun 2023
PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization Yidong Wang Zhuohao Yu Zhengran Zeng Linyi Yang Cunxiang Wang ... Jindong Wang Xingxu Xie Wei Ye Shi-Bo Zhang Yue Zhang ALM ELM 146 249 0 08 Jun 2023
Benchmarking Foundation Models with Language-Model-as-an-Examiner Yushi Bai Jiahao Ying Yixin Cao Xin Lv Yuze He ... Yijia Xiao Haozhe Lyu Jiayin Zhang Juanzi Li Lei Hou ALM ELM 107 149 0 07 Jun 2023
Direct Preference Optimization: Your Language Model is Secretly a Reward Model Rafael Rafailov Archit Sharma E. Mitchell Stefano Ermon Christopher D. Manning Chelsea Finn ALM 393 4,184 0 29 May 2023
G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment Yang Liu Dan Iter Yichong Xu Shuohang Wang Ruochen Xu Chenguang Zhu ELM ALM LM&MA 228 1,215 0 29 Mar 2023