Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates

23 August 2024

Papers citing "Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates"

50 / 67 papers shown

Title
Judging LLMs on a Simplex Patrick Vossler Fan Xia Yifan Mai Jean Feng 20 0 0 28 May 2025
AdaSTaR: Adaptive Data Sampling for Training Self-Taught Reasoners Woosung Koh Wonbeen Oh Jaein Jang MinHyung Lee Hyeongjin Kim Ah Yeon Kim Joonkee Kim Junghyun Lee Taehyeon Kim Se-Young Yun LRM TTA 58 0 0 22 May 2025
TRAIL: Trace Reasoning and Agentic Issue Localization Darshan Deshpande Varun Gangal Hersh Mehta Jitin Krishnan Anand Kannappan Rebecca Qian 73 0 0 13 May 2025
EnronQA: Towards Personalized RAG over Private Documents Michael J. Ryan Danmei Xu Chris Nivera Daniel Campos SILM 86 1 0 01 May 2025
Stay Hungry, Stay Foolish: On the Extended Reading Articles Generation with LLMs Yow-Fu Liou Yu-Chien Tang An-Zi Yen AI4Ed 84 0 0 21 Apr 2025
Benchmarking Multi-National Value Alignment for Large Language Models Chengyi Ju Weijie Shi Chengzhong Liu Yalan Qin Jipeng Zhang ... Jia Zhu Jiajie Xu Yaodong Yang Sirui Han Yike Guo 358 0 0 17 Apr 2025
Fùxì: A Benchmark for Evaluating Language Models on Ancient Chinese Text Understanding and Generation Shangqing Zhao Yuhao Zhou Yupei Ren Zhe Chen Chenghao Jia Fang Zhe Zhaogaung Long Shu Liu Man Lan ALM ELM 111 1 0 20 Mar 2025
No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding Michael Krumdick Charles Lovering Varshini Reddy Seth Ebner Chris Tanner ALM ELM 92 3 0 07 Mar 2025
CodeIF-Bench: Evaluating Instruction-Following Capabilities of Large Language Models in Interactive Code Generation Peiding Wang Lulu Zhang Fang Liu Lin Shi Minxiao Li Bo Shen An Fu ELM LRM 291 1 0 05 Mar 2025
Understand User Opinions of Large Language Models via LLM-Powered In-the-Moment User Experience Interviews Mengqiao Liu Tevin Wang Cassandra A. Cohen Sarah Li Chenyan Xiong LRM 86 0 0 24 Feb 2025
Improve LLM-as-a-Judge Ability as a General Ability Jiachen Yu Shaoning Sun Xiaohui Hu Jiaxu Yan Kaidong Yu Xuelong Li ELM 110 5 0 17 Feb 2025
PlanGenLLMs: A Modern Survey of LLM Planning Capabilities Hui Wei Zihao Zhang Shenghua He Tian Xia Shijia Pan Fei Liu 101 8 0 16 Feb 2025
AI Alignment at Your Discretion Maarten Buyl Hadi Khalaf C. M. Verdun Lucas Monteiro Paes Caio Vieira Machado Flavio du Pin Calmon 69 0 0 10 Feb 2025
Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking Benjamin Feuer Micah Goldblum Teresa Datta Sanjana Nambiar Raz Besaleli Samuel Dooley Max Cembalest John P. Dickerson ALM 101 0 0 28 Jan 2025
Is my Meeting Summary Good? Estimating Quality with a Multi-LLM Evaluator Frederic Kirstein Terry Ruas Bela Gipp 131 2 0 27 Nov 2024
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge Dawei Li Bohan Jiang Liangjie Huang Alimohammad Beigi Chengshuai Zhao ... Canyu Chen Tianhao Wu Kai Shu Lu Cheng Huan Liu ELM AILaw 163 96 0 25 Nov 2024
From Barriers to Tactics: A Behavioral Science-Informed Agentic Workflow for Personalized Nutrition Coaching Eric Yang Tomas Garcia Hannah Williams Bhawesh Kumar Martin Ramé Eileen Rivera Yiran Ma Jonathan Amar Caricia Catalani Yugang Jia OffRL 142 2 0 17 Oct 2024
UCFE: A User-Centric Financial Expertise Benchmark for Large Language Models Yuzhe Yang Yifei Zhang Yan Hu Y. Guo Ruoli Gan ... Haining Wang Qianqian Xie Jimin Huang Honghai Yu Benyou Wang ELM AIFin 59 2 0 17 Oct 2024
Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the data Florian E. Dorner Vivian Y. Nastl Moritz Hardt ELM ALM 64 8 0 17 Oct 2024
JudgeBench: A Benchmark for Evaluating LLM-based Judges Sijun Tan Siyuan Zhuang Kyle Montgomery William Y. Tang Alejandro Cuadron Chenguang Wang Raluca A. Popa Ion Stoica ELM ALM 85 45 0 16 Oct 2024
DHP Benchmark: Are LLMs Good NLG Evaluators? Yicheng Wang Jiayi Yuan Yu-Neng Chuang Zhuoer Wang Yingchi Liu Mark Cusick Param Kulkarni Zhengping Ji Yasser Ibrahim Xia Hu LM&MA ELM 79 3 0 25 Aug 2024
Variational Best-of-N Alignment Afra Amini Tim Vieira Ryan Cotterell Ryan Cotterell BDL 58 20 0 08 Jul 2024
Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges Aman Singh Thakur Kartik Choudhary Venkat Srinik Ramayapally Sankaran Vaidyanathan Dieuwke Hupkes ELM ALM 91 61 0 18 Jun 2024
From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline Tianle Li Wei-Lin Chiang Evan Frick Lisa Dunlap Tianhao Wu Banghua Zhu Joseph E. Gonzalez Ion Stoica ALM 61 144 0 17 Jun 2024
ConSiDERS-The-Human Evaluation Framework: Rethinking Human Evaluation for Generative Large Language Models Aparna Elangovan Ling Liu Lei Xu S. Bodapati Dan Roth ELM 64 9 0 28 May 2024
Large Language Models are Inconsistent and Biased Evaluators Rickard Stureborg Dimitris Alikaniotis Yoshi Suhara ALM 77 60 0 02 May 2024
DPO Meets PPO: Reinforced Token Optimization for RLHF Han Zhong Zikang Shan Guhao Feng Wei Xiong Xinle Cheng Li Zhao Di He Jiang Bian Liwei Wang 108 62 0 29 Apr 2024
LLM Evaluators Recognize and Favor Their Own Generations Arjun Panickssery Samuel R. Bowman Shi Feng 69 172 0 15 Apr 2024
Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators Yann Dubois Balázs Galambosi Percy Liang Tatsunori Hashimoto ALM 76 359 0 06 Apr 2024
The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization Shengyi Huang Michael Noukhovitch Arian Hosseini Kashif Rasul Weixun Wang Lewis Tunstall VLM 44 33 0 24 Mar 2024
RewardBench: Evaluating Reward Models for Language Modeling Nathan Lambert Valentina Pyatkin Jacob Morrison Lester James V. Miranda Bill Yuchen Lin ... Sachin Kumar Tom Zick Yejin Choi Noah A. Smith Hanna Hajishirzi ALM 119 250 0 20 Mar 2024
Improving Reinforcement Learning from Human Feedback Using Contrastive Rewards Wei Shen Xiaoying Zhang Yuanshun Yao Rui Zheng Hongyi Guo Yang Liu ALM 51 13 0 12 Mar 2024
Proxy-RLHF: Decoupling Generation and Alignment in Large Language Model with Proxy Yu Zhu Chuxiong Sun Wenfei Yang Wenqiang Wei Simin Niu ... Zhiyu Li Shifeng Zhang Feiyu Xiong Jie Hu Mingchuan Yang 47 3 0 07 Mar 2024
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference Wei-Lin Chiang Lianmin Zheng Ying Sheng Anastasios Nikolas Angelopoulos Tianle Li ... Hao Zhang Banghua Zhu Michael I. Jordan Joseph E. Gonzalez Ion Stoica OSLM 94 536 0 07 Mar 2024
Direct Language Model Alignment from Online AI Feedback Shangmin Guo Biao Zhang Tianlin Liu Tianqi Liu Misha Khalman ... Thomas Mesnard Yao-Min Zhao Bilal Piot Johan Ferret Mathieu Blondel ALM 54 146 0 07 Feb 2024
Self-Rewarding Language Models Weizhe Yuan Richard Yuanzhe Pang Kyunghyun Cho Xian Li Sainbayar Sukhbaatar Jing Xu Jason Weston ReLM SyDa ALM LRM 285 312 0 18 Jan 2024
Secrets of RLHF in Large Language Models Part II: Reward Modeling Bing Wang Rui Zheng Luyao Chen Yan Liu Shihan Dou ... Qi Zhang Xipeng Qiu Xuanjing Huang Zuxuan Wu Yuanyuan Jiang ALM 75 106 0 11 Jan 2024
On Diversified Preferences of Large Language Model Alignment Dun Zeng Yong Dai Pengyu Cheng Longyue Wang Tianhao Hu Wanshun Chen Nan Du Zenglin Xu ALM 57 16 0 12 Dec 2023
Sample Efficient Preference Alignment in LLMs via Active Exploration Viraj Mehta Vikramjeet Das Ojash Neopane Yijia Dai Ilija Bogunovic Ilija Bogunovic Willie Neiswanger Stefano Ermon Jeff Schneider Willie Neiswanger OffRL 74 11 0 01 Dec 2023
Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Game Pengyu Cheng Yifan Yang Jian Li Yong Dai Tianhao Hu Peixin Cao Nan Du Xiaolong Li 40 29 0 14 Nov 2023
JudgeLM: Fine-tuned Large Language Models are Scalable Judges Lianghui Zhu Xinggang Wang Xinlong Wang ELM ALM 87 125 0 26 Oct 2023
Verbosity Bias in Preference Labeling by Large Language Models Keita Saito Akifumi Wachi Koki Wataoka Youhei Akimoto ALM 37 32 0 16 Oct 2023
Generative Judge for Evaluating Alignment Junlong Li Shichao Sun Weizhe Yuan Run-Ze Fan Hai Zhao Pengfei Liu ELM ALM 44 81 0 09 Oct 2023
Stabilizing RLHF through Advantage Model and Selective Rehearsal Baolin Peng Linfeng Song Ye Tian Lifeng Jin Haitao Mi Dong Yu 50 19 0 18 Sep 2023
Llama 2: Open Foundation and Fine-Tuned Chat Models Hugo Touvron Louis Martin Kevin R. Stone Peter Albert Amjad Almahairi ... Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov Thomas Scialom AI4MH ALM 206 11,636 0 18 Jul 2023
Secrets of RLHF in Large Language Models Part I: PPO Rui Zheng Shihan Dou Songyang Gao Yuan Hua Wei Shen ... Hang Yan Tao Gui Qi Zhang Xipeng Qiu Xuanjing Huang ALM OffRL 61 163 0 11 Jul 2023
Style Over Substance: Evaluation Biases for Large Language Models Minghao Wu Alham Fikri Aji ALM ELM 67 44 0 06 Jul 2023
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena Lianmin Zheng Wei-Lin Chiang Ying Sheng Siyuan Zhuang Zhanghao Wu ... Dacheng Li Eric Xing Haotong Zhang Joseph E. Gonzalez Ion Stoica ALM OSLM ELM 242 4,186 0 09 Jun 2023
PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization Yidong Wang Zhuohao Yu Zhengran Zeng Linyi Yang Cunxiang Wang ... Jindong Wang Xingxu Xie Wei Ye Shi-Bo Zhang Yue Zhang ALM ELM 85 242 0 08 Jun 2023
Benchmarking Foundation Models with Language-Model-as-an-Examiner Yushi Bai Jiahao Ying Yixin Cao Xin Lv Yuze He ... Yijia Xiao Haozhe Lyu Jiayin Zhang Juanzi Li Lei Hou ALM ELM 58 141 0 07 Jun 2023