Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges

18 June 2024

Aman Singh Thakur

Kartik Choudhary

Venkat Srinik Ramayapally

Sankaran Vaidyanathan

Papers citing "Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges"

50 / 52 papers shown

Title
clem:todd: A Framework for the Systematic Benchmarking of LLM-Based Task-Oriented Dialogue System Realisations Chalamalasetti Kranti Sherzod Hakimov David Schlangen LLMAG 42 0 0 08 May 2025
Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems Shaokun Zhang Ming Yin Jieyu Zhang J. H. Liu Zhiguang Han ... Beibin Li Chi Wang H. Wang Y. Chen Qingyun Wu 49 0 0 30 Apr 2025
Anyprefer: An Agentic Framework for Preference Data Synthesis Yiyang Zhou Z. Wang Tianle Wang Shangyu Xing Peng Xia ... Chetan Bansal Weitong Zhang Ying Wei Mohit Bansal Huaxiu Yao 61 0 0 27 Apr 2025
Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments Y. Li Jama Hussein Mohamud Chongren Sun Di Wu Benoit Boulet LLMAG ELM 70 0 0 23 Apr 2025
Multi-Stage Retrieval for Operational Technology Cybersecurity Compliance Using Large Language Models: A Railway Casestudy Regan Bolton Mohammadreza Sheikhfathollahi Simon Parkinson Dan Basher Howard Parkinson 34 0 0 18 Apr 2025
LLM-as-a-Judge: Reassessing the Performance of LLMs in Extractive QA Xanh Ho Jiahao Huang Florian Boudin Akiko Aizawa ELM 31 0 0 16 Apr 2025
MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languages Dieuwke Hupkes Nikolay Bogoychev 121 0 0 14 Apr 2025
Multi-Agent LLM Judge: automatic personalized LLM judge design for evaluating natural language generation applications Hongliu Cao Ilias Driouich Robin Singh Eoin Thomas ELM 36 0 0 01 Apr 2025
KOFFVQA: An Objectively Evaluated Free-form VQA Benchmark for Large Vision-Language Models in the Korean Language Yoonshik Kim Jaeyoon Jung 37 0 0 31 Mar 2025
Validating LLM-as-a-Judge Systems in the Absence of Gold Labels Luke M. Guerdan Solon Barocas Kenneth Holstein Hanna M. Wallach Zhiwei Steven Wu Alexandra Chouldechova ALM ELM 203 0 0 13 Mar 2025
Safer or Luckier? LLMs as Safety Evaluators Are Not Robust to Artifacts Hongyu Chen Seraphina Goldfarb-Tarrant 45 0 0 12 Mar 2025
DAFE: LLM-Based Evaluation Through Dynamic Arbitration for Free-Form Question-Answering Sher Badshah Hassan Sajjad 60 1 0 11 Mar 2025
Learning and generalization of robotic dual-arm manipulation of boxes from demonstrations via Gaussian Mixture Models (GMMs) Qian Ying Lee Suhas Raghavendra Kulkarni Kenzhi Iskandar Wong Lin Yang Bernardo Noronha Yongjun Wee Tzu-Yi Hung Domenico Campolo 45 0 0 07 Mar 2025
Revitalizing Saturated Benchmarks: A Weighted Metric Approach for Differentiating Large Language Model Performance Bryan Etzine Masoud Hashemi Nishanth Madhusudhan Sagar Davasam Roshnee Sharma Sathwik Tejaswi Madhusudhan Vikas Yadav 39 0 0 07 Mar 2025
LLMs Can Generate a Better Answer by Aggregating Their Own Responses Zichong Li Xinyu Feng Yuheng Cai Zixuan Zhang Tianyi Liu Chen Liang Weizhu Chen Haoyu Wang T. Zhao LRM 50 1 0 06 Mar 2025
SEOE: A Scalable and Reliable Semantic Evaluation Framework for Open Domain Event Detection Yi-Fan Lu Xian-Ling Mao Tian Lan Tong Zhang Yu-Shi Zhu Heyan Huang 52 0 0 05 Mar 2025
NUTSHELL: A Dataset for Abstract Generation from Scientific Talks Maike Züfle Sara Papi Beatrice Savoldi Marco Gaido L. Bentivogli Jan Niehues 38 0 0 24 Feb 2025
Proactive Privacy Amnesia for Large Language Models: Safeguarding PII with Negligible Impact on Model Utility Martin Kuo Jingyang Zhang Jianyi Zhang Minxue Tang Louis DiValentin ... William Chen Amin Hass Tianlong Chen Y. Chen H. Li MU KELM 37 2 0 24 Feb 2025
Correlating and Predicting Human Evaluations of Language Models from Natural Language Processing Benchmarks Rylan Schaeffer Punit Singh Koura Binh Tang R. Subramanian Aaditya K. Singh ... Vedanuj Goswami Sergey Edunov Dieuwke Hupkes Sanmi Koyejo Sharan Narang ALM 69 0 0 24 Feb 2025
Multi-Attribute Steering of Language Models via Targeted Intervention Duy Nguyen Archiki Prasad Elias Stengel-Eskin Mohit Bansal LLMSV 110 0 0 18 Feb 2025
Towards Reasoning Ability of Small Language Models Gaurav Srivastava Shuxiang Cao Xuan Wang ReLM LRM 49 4 0 17 Feb 2025
An Empirical Analysis of Uncertainty in Large Language Model Evaluations Qiujie Xie Qingqiu Li Zhuohao Yu Yuejie Zhang Yue Zhang Linyi Yang ELM 63 1 0 15 Feb 2025
Combining Large Language Models with Static Analyzers for Code Review Generation Imen Jaoua Oussama Ben Sghaier Houari Sahraoui 58 1 0 10 Feb 2025
Preference Leakage: A Contamination Problem in LLM-as-a-judge Dawei Li Renliang Sun Yue Huang Ming Zhong Bohan Jiang J. Han X. Zhang Wei Wang Huan Liu 65 11 0 03 Feb 2025
LLM-Powered Benchmark Factory: Reliable, Generic, and Efficient Peiwen Yuan Shaoxiong Feng Yiwei Li X. U. Wang Y. Zhang Jiayi Shi Chuyi Tan Boyuan Pan Yao Hu Kan Li 68 2 0 02 Feb 2025
From Cool Demos to Production-Ready FMware: Core Challenges and a Technology Roadmap Gopi Krishnan Rajbahadur G. Oliva Dayi Lin Ahmed E. Hassan 41 1 0 28 Jan 2025
WarriorCoder: Learning from Expert Battles to Augment Code Large Language Models Huawen Feng Pu Zhao Qingfeng Sun Can Xu Fangkai Yang ... Qianli Ma Qingwei Lin Saravan Rajmohan Dongmei Zhang Qi Zhang AAML ALM 62 0 0 23 Dec 2024
Outcome-Refining Process Supervision for Code Generation Zhuohao Yu Weizheng Gu Yidong Wang Zhengran Zeng Jindong Wang Wei Ye Shikun Zhang LRM 89 4 0 19 Dec 2024
KARRIEREWEGE: A Large Scale Career Path Prediction Dataset Elena Senger Yuri Campbell Rob van der Goot Barbara Plank AI4TS 78 1 0 19 Dec 2024
JuStRank: Benchmarking LLM Judges for System Ranking Ariel Gera Odellia Boni Yotam Perlitz Roy Bar-Haim Lilach Eden Asaf Yehudai ALM ELM 98 3 0 12 Dec 2024
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge Dawei Li Bohan Jiang Liangjie Huang Alimohammad Beigi Chengshuai Zhao ... Canyu Chen Tianhao Wu Kai Shu Lu Cheng Huan Liu ELM AILaw 120 65 0 25 Nov 2024
Do LLMs Agree on the Creativity Evaluation of Alternative Uses? Abdullah Al Rabeyah Fabrício Góes Marco Volpe Talles Medeiros 77 1 0 23 Nov 2024
Bayesian Calibration of Win Rate Estimation with LLM Evaluators Yicheng Gao G. Xu Zhe Wang Arman Cohan 38 6 0 07 Nov 2024
Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluation Dongryeol Lee Yerin Hwang Yongil Kim Joonsuk Park Kyomin Jung ELM 70 5 0 28 Oct 2024
From Barriers to Tactics: A Behavioral Science-Informed Agentic Workflow for Personalized Nutrition Coaching Eric Yang Tomas Garcia Hannah Williams Bhawesh Kumar Martin Ramé Eileen Rivera Yiran Ma Jonathan Amar Caricia Catalani Yugang Jia OffRL 32 2 0 17 Oct 2024
MIRAGE-Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systems Nandan Thakur Suleman Kazi Ge Luo Jimmy J. Lin Amin Ahmad VLM RALM 28 7 0 17 Oct 2024
Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the data Florian E. Dorner Vivian Y. Nastl Moritz Hardt ELM ALM 42 5 0 17 Oct 2024
Cognitive Overload Attack:Prompt Injection for Long Context Bibek Upadhayay Vahid Behzadan Amin Karbasi AAML 28 2 0 15 Oct 2024
SecCodePLT: A Unified Platform for Evaluating the Security of Code GenAI Yu Yang Yuzhou Nie Zhun Wang Yuheng Tang Wenbo Guo Bo Li D. Song ELM 38 6 0 14 Oct 2024
Agent-as-a-Judge: Evaluate Agents with Agents Mingchen Zhuge Changsheng Zhao Dylan R. Ashley Wenyi Wang Dmitrii Khizbullin ... Raghuraman Krishnamoorthi Yuandong Tian Yangyang Shi Vikas Chandra Jürgen Schmidhuber ELM 60 34 0 14 Oct 2024
JurEE not Judges: safeguarding llm interactions with small, specialised Encoder Ensembles Dom Nasrabadi 31 1 0 11 Oct 2024
Why context matters in VQA and Reasoning: Semantic interventions for VLM input modalities Kenza Amara Lukas Klein Carsten T. Lüth Paul Jäger Hendrik Strobelt Mennatallah El-Assady 25 1 0 02 Oct 2024
Archon: An Architecture Search Framework for Inference-Time Techniques Jon Saad-Falcon Adrian Gamarra Lafuente Shlok Natarajan Nahum Maru Hristo Todorov ... E. Kelly Buchanan Mayee Chen Neel Guha Christopher Ré Azalia Mirhoseini AI4CE 33 19 0 23 Sep 2024
Model-in-the-Loop (MILO): Accelerating Multimodal AI Data Annotation with LLMs Yifan Wang David Stevens Pranay Shah Wenwen Jiang Miao Liu ... Boying Gong Daniel Lee Jiabo Hu Ning Zhang Bob Kamma 40 1 0 16 Sep 2024
ConsistencyTrack: A Robust Multi-Object Tracker with a Generation Strategy of Consistency Model Lifan Jiang Zhihui Wang Siqi Yin Guangxiao Ma Peng Zhang Boxi Wu DiffM 51 0 0 28 Aug 2024
Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates Hui Wei Shenghua He Tian Xia Andy H. Wong Jingyang Lin Mei Han Mei Han ALM ELM 61 23 0 23 Aug 2024
Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form Text Sher Badshah Hassan Sajjad ELM 38 9 0 17 Aug 2024
Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement Jaehun Jung Faeze Brahman Yejin Choi ALM 42 12 0 25 Jul 2024
PM-LLM-Benchmark: Evaluating Large Language Models on Process Mining Tasks Alessandro Berti Humam Kourani Wil M.P. van der Aalst 34 6 0 18 Jul 2024
JudgeLM: Fine-tuned Large Language Models are Scalable Judges Lianghui Zhu Xinggang Wang Xinlong Wang ELM ALM 56 106 0 26 Oct 2023