JudgeBench: A Benchmark for Evaluating LLM-based Judges

16 October 2024

Ion Stoica

Papers citing "JudgeBench: A Benchmark for Evaluating LLM-based Judges"

25 / 25 papers shown

Title
HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages Zihan Wang Jiaqi Zeng Olivier Delalleau Hoo-Chang Shin Felipe Soares Alexander Bukharin Ellie Evans Yi Dong Oleksii Kuchaiev 22 0 0 16 May 2025
J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning Chenxi Whitehouse Tianlu Wang Ping Yu Xian Li Jason Weston Ilia Kulikov Swarnadeep Saha ALM ELM LRM 19 0 0 15 May 2025
WixQA: A Multi-Dataset Benchmark for Enterprise Retrieval-Augmented Generation Dvir Cohen Lin Burg Sviatoslav Pykhnivskyi Hagit Gur Stanislav Kovynov Olga Atzmon Gilad Barkan RALM 26 0 0 13 May 2025
Thoughts without Thinking: Reconsidering the Explanatory Value of Chain-of-Thought Reasoning in LLMs through Agentic Pipelines R. Manuvinakurike Emanuel Moss E. A. Watkins Saurav Sahay G. Raffa L. Nachman LRM 31 0 0 01 May 2025
Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems Shaokun Zhang Ming Yin Jieyu Zhang Jing Liu Zhiguang Han ... Beibin Li Chi Wang H. Wang Yuxiao Chen Qingyun Wu 49 1 0 30 Apr 2025
Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments Y. Li Jama Hussein Mohamud Chongren Sun Di Wu Benoit Boulet LLMAG ELM 72 0 0 23 Apr 2025
Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators Yilun Zhou Austin Xu Peifeng Wang Caiming Xiong Chenyu You ELM ALM LRM 53 2 0 21 Apr 2025
Efficient MAP Estimation of LLM Judgment Performance with Prior Transfer Huaizhi Qu Inyoung Choi Zhen Tan Song Wang Sukwon Yun Qi Long Faizan Siddiqui Kwonjoon Lee Tianlong Chen 43 0 0 17 Apr 2025
A Multi-Model Adaptation of Speculative Decoding for Classification Somnath Roy Padharthi Sreekar Srivatsa Narasimha Anubhav Anand 43 0 0 23 Mar 2025
Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settings Austin Xu Srijan Bansal Yifei Ming Semih Yavuz Chenyu You ELM 95 3 0 19 Mar 2025
UPME: An Unsupervised Peer Review Framework for Multimodal Large Language Model Evaluation Qihui Zhang Munan Ning Zheyuan Liu Yanbo Wang Jiayi Ye Yue Huang Shuo Yang Xiao Chen Y. Song Li Yuan LRM 61 0 0 19 Mar 2025
ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learning Bo Liu Yunxiang Li Yangqiu Song Hanjing Wang Linyi Yang Mark W. Schmidt Jun Wang Weinan Zhang Shuyue Hu Ying Wen LLMAG KELM LRM AI4CE 92 6 0 12 Mar 2025
GRP: Goal-Reversed Prompting for Zero-Shot Evaluation with LLMs Mingyang Song Mao Zheng Xuan Luo LRM 60 0 0 08 Mar 2025
No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding Michael Krumdick Charles Lovering Varshini Reddy Seth Ebner Chris Tanner ALM ELM 55 2 0 07 Mar 2025
Improving LLM-as-a-Judge Inference with the Judgment Distribution Victor Wang Michael J.Q. Zhang Eunsol Choi 58 1 0 04 Mar 2025
LangProBe: a Language Programs Benchmark Shangyin Tan Lakshya A Agrawal Arnav Singhvi Liheng Lai Michael J Ryan Dan Klein Omar Khattab Koushik Sen Matei A. Zaharia 64 0 0 27 Feb 2025
Can Large Language Models Detect Errors in Long Chain-of-Thought Reasoning? Yancheng He Shilong Li Xiaozhong Liu Weixun Wang Xingyuan Bu ... Zhongyuan Peng Zhenru Zhang Zhicheng Zheng Wenbo Su Bo Zheng ELM LRM 86 8 0 26 Feb 2025
Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems Hao Peng Y. Qi Xiaozhi Wang Zijun Yao Bin Xu Lei Hou Juanzi Li ALM LRM 62 4 0 26 Feb 2025
CodeCriticBench: A Holistic Code Critique Benchmark for Large Language Models Alexander Zhang Marcus Dong Jing Liu Wei Zhang Yejie Wang ... Yancheng He K. Deng Wangchunshu Zhou Wenhao Huang Zhenru Zhang LRM 62 2 0 23 Feb 2025
Critical-Questions-of-Thought: Steering LLM reasoning with Argumentative Querying Federico Castagna I. Sassoon Simon Parsons LRM 85 0 0 19 Dec 2024
JuStRank: Benchmarking LLM Judges for System Ranking Ariel Gera Odellia Boni Yotam Perlitz Roy Bar-Haim Lilach Eden Asaf Yehudai ALM ELM 98 3 0 12 Dec 2024
VLRewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models Lei Li Y. X. Wei Zhihui Xie Xuqing Yang Yifan Song ... Tianyu Liu Sujian Li Bill Yuchen Lin Lingpeng Kong Qiang Liu CoGe VLM 120 25 0 26 Nov 2024
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge Dawei Li Bohan Jiang Liangjie Huang Alimohammad Beigi Chengshuai Zhao ... Canyu Chen Tianhao Wu Kai Shu Lu Cheng Huan Liu ELM AILaw 123 70 0 25 Nov 2024
RevisEval: Improving LLM-as-a-Judge via Response-Adapted References Qiyuan Zhang Yufei Wang Tiezheng YU Yuxin Jiang Chuhan Wu ... Xin Jiang Lifeng Shang Ruiming Tang Fuyuan Lyu Chen Ma 31 4 0 07 Oct 2024
DHP Benchmark: Are LLMs Good NLG Evaluators? Yicheng Wang Jiayi Yuan Yu-Neng Chuang Zhuoer Wang Yingchi Liu Mark Cusick Param Kulkarni Zhengping Ji Yasser Ibrahim Xia Hu LM&MA ELM 49 3 0 25 Aug 2024