JudgeLM: Fine-tuned Large Language Models are Scalable Judges

26 October 2023

Papers citing "JudgeLM: Fine-tuned Large Language Models are Scalable Judges"

50 / 92 papers shown

Title
Benchmarking Critical Questions Generation: A Challenging Reasoning Task for Large Language Models Banca Calvo Figueras Rodrigo Agerri ALM ELM LRM 19 0 0 16 May 2025
TRAIL: Trace Reasoning and Agentic Issue Localization Darshan Deshpande Varun Gangal Hersh Mehta Jitin Krishnan Anand Kannappan Rebecca Qian 25 0 0 13 May 2025
SAS-Bench: A Fine-Grained Benchmark for Evaluating Short Answer Scoring with Large Language Models Peichao Lai Kaipeng Zhang Yi Lin L. Zhang Feiyang Ye ... Yanwei Xu Conghui He Yixuan Wang Wentao Zhang Bin Cui ELM LRM 47 0 0 12 May 2025
To Judge or not to Judge: Using LLM Judgements for Advertiser Keyphrase Relevance at eBay Soumik Dey Hansi Wu Binbin Li 45 0 0 07 May 2025
am-ELO: A Stable Framework for Arena-based LLM Evaluation Zirui Liu Jiatong Li Yan Zhuang Qiang Liu Shuanghong Shen Jie Ouyang Mingyue Cheng Shijin Wang 41 0 0 06 May 2025
Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Models Bang Zhang Ruotian Ma Qingxuan Jiang Peisong Wang Jiaqi Chen ... Fanghua Ye Jian Li Yifan Yang Zhaopeng Tu Xiaolong Li LLMAG ELM ALM 109 0 1 01 May 2025
Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks Yixin Cao Shibo Hong Xuzhao Li Jiahao Ying Yubo Ma ... Juanzi Li Aixin Sun Xuanjing Huang Tat-Seng Chua Tianwei Zhang ALM ELM 86 2 0 26 Apr 2025
Process Reward Models That Think Muhammad Khalifa Rishabh Agarwal Lajanugen Logeswaran Jaekyeom Kim Hao Peng Moontae Lee Honglak Lee Lu Wang OffRL ALM LRM 44 1 0 23 Apr 2025
PROMPTEVALS: A Dataset of Assertions and Guardrails for Custom Production Large Language Model Pipelines Reya Vir Shreya Shankar Harrison Chase Will Fu-Hinthorn Aditya G. Parameswaran AI4TS 32 0 0 20 Apr 2025
xVerify: Efficient Answer Verifier for Reasoning Model Evaluations Ding Chen Qingchen Yu P. Wang W. Zhang Bo Tang Feiyu Xiong X. Li Minchuan Yang Z. Li ALM LRM 36 2 0 14 Apr 2025
AgentAda: Skill-Adaptive Data Analytics for Tailored Insight Discovery Amirhossein Abaskohi A. Ramesh Shailesh Nanisetty Chirag Goel David Vazquez Christopher Pal Spandana Gella Giuseppe Carenini I. Laradji 39 0 0 10 Apr 2025
TALE: A Tool-Augmented Framework for Reference-Free Evaluation of Large Language Models Sher Badshah Ali Emami Hassan Sajjad LLMAG ELM 45 0 0 10 Apr 2025
Benchmarking Adversarial Robustness to Bias Elicitation in Large Language Models: Scalable Automated Assessment with LLM-as-a-Judge Riccardo Cantini A. Orsino Massimo Ruggiero Domenico Talia AAML ELM 45 0 0 10 Apr 2025
Sequential-NIAH: A Needle-In-A-Haystack Benchmark for Extracting Sequential Needles from Long Contexts Yifei Yu Qian Zhang Lingfeng Qiao Di Yin Fang Li Jie Wang Z. Chen Suncong Zheng Xiaolong Liang Xingchen Sun 39 0 0 07 Apr 2025
Taxonomy-Aware Evaluation of Vision-Language Models Vésteinn Snæbjarnarson Kevin Du Niklas Stoehr Serge J. Belongie Ryan Cotterell Nico Lang Stella Frank 32 0 0 07 Apr 2025
Do LLM Evaluators Prefer Themselves for a Reason? Wei-Lin Chen Zhepei Wei Xinyu Zhu Shi Feng Yu Meng ELM LRM 42 0 0 04 Apr 2025
Prompt, Divide, and Conquer: Bypassing Large Language Model Safety Filters via Segmented and Distributed Prompt Processing Johan Wahréus Ahmed Mohamed Hussain P. Papadimitratos 58 0 0 27 Mar 2025
A Multi-Model Adaptation of Speculative Decoding for Classification Somnath Roy Padharthi Sreekar Srivatsa Narasimha Anubhav Anand 41 0 0 23 Mar 2025
REPA: Russian Error Types Annotation for Evaluating Text Generation and Judgment Capabilities Alexander Pugachev Alena Fenogenova Vladislav Mikhailov Ekaterina Artemova 37 0 0 17 Mar 2025
GRP: Goal-Reversed Prompting for Zero-Shot Evaluation with LLMs Mingyang Song Mao Zheng Xuan Luo LRM 60 0 0 08 Mar 2025
Dynamic Knowledge Integration for Evidence-Driven Counter-Argument Generation with Large Language Models Anar Yeginbergen Maite Oronoz Rodrigo Agerri 53 0 0 07 Mar 2025
RocketEval: Efficient Automated LLM Evaluation via Grading Checklist Tianjun Wei Wei Wen Ruizhi Qiao Xing Sun Jianghong Ma ALM ELM 50 1 0 07 Mar 2025
Is Your Video Language Model a Reliable Judge? M. Liu Wensheng Zhang 64 2 0 07 Mar 2025
Benchmarking Large Language Models on Multiple Tasks in Bioinformatics NLP with Prompting Jiyue Jiang Pengan Chen J. T. Wang Dongchen He Ziqin Wei ... Yimin Fan Xiangyu Shi Jiashuo Sun Chuan Wu Y. Li LM&MA 50 0 0 06 Mar 2025
Improving LLM-as-a-Judge Inference with the Judgment Distribution Victor Wang Michael J.Q. Zhang Eunsol Choi 55 0 0 04 Mar 2025
OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scale Haoyang Li Shang Wu Xiaokang Zhang Xinmei Huang J. Zhang ... Tieying Zhang Jianjun Chen Rui Shi H. Chen C. Li SyDa 80 1 0 04 Mar 2025
Argument Summarization and its Evaluation in the Era of Large Language Models Moritz Altemeyer Steffen Eger Johannes Daxenberger Tim Altendorf Philipp Cimiano Benjamin Schiller LM&MA ELM LRM 67 0 0 02 Mar 2025
Learning to Align Multi-Faceted Evaluation: A Unified and Robust Framework Kaishuai Xu Tiezheng YU Wenjun Hou Yi Cheng Liangyou Li Xin Jiang Lifeng Shang Qiang Liu Wenjie Li ELM 66 0 0 26 Feb 2025
Savaal: Scalable Concept-Driven Question Generation to Enhance Human Learning Kimia Noorbakhsh Joseph Chandler Pantea Karimi M. Alizadeh H. Balakrishnan LRM 46 1 0 18 Feb 2025
Combining Large Language Models with Static Analyzers for Code Review Generation Imen Jaoua Oussama Ben Sghaier Houari Sahraoui 65 1 0 10 Feb 2025
DeepThink: Aligning Language Models with Domain-Specific User Intents Yang Li Mingxuan Luo Yeyun Gong Chen Lin Jian Jiao Yi Liu Kaili Huang LRM ALM ELM 56 0 0 08 Feb 2025
Synthetic Data Can Mislead Evaluations: Membership Inference as Machine Text Detection Ali Naseh Niloofar Mireshghallah 58 0 0 20 Jan 2025
Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models Yulei Qin Yuncheng Yang Pengcheng Guo Gang Li Hang Shao Yuchen Shi Zihan Xu Yun Gu Ke Li Xing Sun ALM 93 12 0 31 Dec 2024
The Superalignment of Superhuman Intelligence with Large Language Models Minlie Huang Yingkang Wang Shiyao Cui Pei Ke J. Tang 113 1 0 15 Dec 2024
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge Dawei Li Bohan Jiang Liangjie Huang Alimohammad Beigi Chengshuai Zhao ... Canyu Chen Tianhao Wu Kai Shu Lu Cheng Huan Liu ELM AILaw 120 67 0 25 Nov 2024
VERITAS: A Unified Approach to Reliability Evaluation Rajkumar Ramamurthy Meghana Arakkal Rajeev Oliver Molenschot James Zou Nazneen Rajani HILM 47 1 0 05 Nov 2024
Rate, Explain and Cite (REC): Enhanced Explanation and Attribution in Automatic Evaluation by Large Language Models Aliyah R. Hsu James Zhu Zhichao Wang Bin Bi Shubham Mehrotra ... Sougata Chaudhuri Regunathan Radhakrishnan S. Asur Claire Na Cheng Bin Yu ALM LRM 69 0 0 03 Nov 2024
Provenance: A Light-weight Fact-checker for Retrieval Augmented LLM Generation Output Hithesh Sankararaman Mohammed Nasheed Yasin Tanner Sorensen Alessandro Di Bari Andreas Stolcke HILM 30 0 0 01 Nov 2024
Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluation Dongryeol Lee Yerin Hwang Yongil Kim Joonsuk Park Kyomin Jung ELM 72 5 0 28 Oct 2024
CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolution Maosong Cao Alexander Lam Haodong Duan Hongwei Liu S. Zhang Kai Chen AILaw ELM 42 13 0 21 Oct 2024
DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processing Shreya Shankar Tristan Chambers Eugene Wu Aditya G. Parameswaran Eugene Wu LLMAG 56 6 0 16 Oct 2024
JudgeBench: A Benchmark for Evaluating LLM-based Judges Sijun Tan Siyuan Zhuang Kyle Montgomery William Y. Tang Alejandro Cuadron Chenguang Wang Raluca A. Popa Ion Stoica ELM ALM 51 38 0 16 Oct 2024
RevisEval: Improving LLM-as-a-Judge via Response-Adapted References Qiyuan Zhang Yufei Wang Tiezheng YU Yuxin Jiang Chuhan Wu ... Xin Jiang Lifeng Shang Ruiming Tang Fuyuan Lyu Chen Ma 31 4 0 07 Oct 2024
The Visualization JUDGE : Can Multimodal Foundation Models Guide Visualization Design Through Visual Perception? Matthew Berger Shusen Liu 31 1 0 05 Oct 2024
Better Instruction-Following Through Minimum Bayes Risk Ian Wu Patrick Fernandes Amanda Bertsch Seungone Kim Sina Pakazad Graham Neubig 48 9 0 03 Oct 2024
Mitigating the Bias of Large Language Model Evaluation Hongli Zhou Hui Huang Yunfei Long Bing Xu Conghui Zhu Hailong Cao Muyun Yang Tiejun Zhao ELM 14 2 0 25 Sep 2024
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models Haoran Que Feiyu Duan Liqun He Yutao Mou Wangchunshu Zhou ... Ge Zhang Junran Peng Zhaoxiang Zhang Songyang Zhang Kai Chen LM&MA ELM VLM 51 11 0 24 Sep 2024
FLEX: Expert-level False-Less EXecution Metric for Reliable Text-to-SQL Benchmark Heegyu Kim Taeyang Jeon Seunghwan Choi Seungtaek Choi Hyunsouk Cho 47 0 0 24 Sep 2024
GroUSE: A Benchmark to Evaluate Evaluators in Grounded Question Answering Sacha Muller António Loison Bilel Omrani Gautier Viaud RALM ELM 38 1 0 10 Sep 2024
What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluation Dingyi Yang Qin Jin 44 5 0 26 Aug 2024