v1v2 (latest)

JudgeLM: Fine-tuned Large Language Models are Scalable Judges

26 October 2023

Papers citing "JudgeLM: Fine-tuned Large Language Models are Scalable Judges"

50 / 110 papers shown

Title
DeepThink: Aligning Language Models with Domain-Specific User Intents Yang Li Mingxuan Luo Yeyun Gong Chen Lin Jian Jiao Yi Liu Kaili Huang LRM ALM ELM 136 0 0 08 Feb 2025
Synthetic Data Can Mislead Evaluations: Membership Inference as Machine Text Detection Ali Naseh Niloofar Mireshghallah 119 0 0 20 Jan 2025
Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models Yulei Qin Yuncheng Yang Pengcheng Guo Gang Li Hang Shao Yuchen Shi Zihan Xu Yun Gu Ke Li Xing Sun ALM 209 13 0 31 Dec 2024
The Superalignment of Superhuman Intelligence with Large Language Models Minlie Huang Yingkang Wang Shiyao Cui Pei Ke J. Tang 176 1 0 15 Dec 2024
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge Dawei Li Bohan Jiang Liangjie Huang Alimohammad Beigi Chengshuai Zhao ... Canyu Chen Tianhao Wu Kai Shu Lu Cheng Huan Liu ELM AILaw 370 112 0 25 Nov 2024
VERITAS: A Unified Approach to Reliability Evaluation Rajkumar Ramamurthy Meghana Arakkal Rajeev Oliver Molenschot James Zou Nazneen Rajani HILM 101 1 0 05 Nov 2024
Rate, Explain and Cite (REC): Enhanced Explanation and Attribution in Automatic Evaluation by Large Language Models Aliyah R. Hsu James Zhu Zhichao Wang Bin Bi Shubham Mehrotra ... Sougata Chaudhuri Regunathan Radhakrishnan S. Asur Claire Na Cheng Bin Yu ALM LRM 184 0 0 03 Nov 2024
Provenance: A Light-weight Fact-checker for Retrieval Augmented LLM Generation Output Hithesh Sankararaman Mohammed Nasheed Yasin Tanner Sorensen Alessandro Di Bari Andreas Stolcke HILM 35 1 0 01 Nov 2024
Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluation Dongryeol Lee Yerin Hwang Yongil Kim Joonsuk Park Kyomin Jung ELM 155 10 0 28 Oct 2024
CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolution Maosong Cao Alexander Lam Haodong Duan Hongwei Liu Shanghang Zhang Kai Chen AILaw ELM 95 20 0 21 Oct 2024
DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processing Shreya Shankar Tristan Chambers Eugene Wu Aditya G. Parameswaran Eugene Wu LLMAG 152 9 0 16 Oct 2024
JudgeBench: A Benchmark for Evaluating LLM-based Judges Sijun Tan Siyuan Zhuang Kyle Montgomery William Y. Tang Alejandro Cuadron Chenguang Wang Raluca A. Popa Ion Stoica ELM ALM 155 52 0 16 Oct 2024
RevisEval: Improving LLM-as-a-Judge via Response-Adapted References Qiyuan Zhang Yufei Wang Tiezheng YU Yuxin Jiang Chuhan Wu ... Xin Jiang Lifeng Shang Ruiming Tang Fuyuan Lyu Chen Ma 124 7 0 07 Oct 2024
The Visualization JUDGE : Can Multimodal Foundation Models Guide Visualization Design Through Visual Perception? Matthew Berger Shusen Liu 77 1 0 05 Oct 2024
Better Instruction-Following Through Minimum Bayes Risk Ian Wu Patrick Fernandes Amanda Bertsch Seungone Kim Sina Pakazad Graham Neubig 141 11 0 03 Oct 2024
Mitigating the Bias of Large Language Model Evaluation Hongli Zhou Hui Huang Yunfei Long Bing Xu Conghui Zhu Hailong Cao Muyun Yang Tiejun Zhao ELM 52 3 0 25 Sep 2024
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models Haoran Que Feiyu Duan Liqun He Yutao Mou Wangchunshu Zhou ... Ge Zhang Junran Peng Zhaoxiang Zhang Songyang Zhang Kai Chen LM&MA ELM VLM 106 16 0 24 Sep 2024
FLEX: Expert-level False-Less EXecution Metric for Reliable Text-to-SQL Benchmark Heegyu Kim Taeyang Jeon Seunghwan Choi Seungtaek Choi Hyunsouk Cho 127 0 0 24 Sep 2024
GroUSE: A Benchmark to Evaluate Evaluators in Grounded Question Answering Sacha Muller António Loison Bilel Omrani Gautier Viaud RALM ELM 110 2 0 10 Sep 2024
What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluation Dingyi Yang Qin Jin 130 7 0 26 Aug 2024
DHP Benchmark: Are LLMs Good NLG Evaluators? Yicheng Wang Jiayi Yuan Yu-Neng Chuang Zhuoer Wang Yingchi Liu Mark Cusick Param Kulkarni Zhengping Ji Yasser Ibrahim Xia Hu LM&MA ELM 123 4 0 25 Aug 2024
Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates Hui Wei Shenghua He Tian Xia Andy H. Wong Jingyang Lin Mei Han Mei Han ALM ELM 194 32 0 23 Aug 2024
Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form Text Sher Badshah Hassan Sajjad ELM 98 14 0 17 Aug 2024
Decoding Biases: Automated Methods and LLM Judges for Gender Bias Detection in Language Models Shachi H. Kumar Saurav Sahay Sahisnu Mazumder Eda Okur R. Manuvinakurike Nicole Beckage Hsuan Su Hung-yi Lee L. Nachman ELM 99 18 0 07 Aug 2024
Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement Jaehun Jung Faeze Brahman Yejin Choi ALM 93 23 0 25 Jul 2024
PrimeGuard: Safe and Helpful LLMs through Tuning-Free Routing Blazej Manczak Eliott Zemour Eric Lin Vaikkunth Mugunthan 48 4 0 23 Jul 2024
Lynx: An Open Source Hallucination Evaluation Model Selvan Sunitha Ravi B. Mielczarek Anand Kannappan Douwe Kiela Rebecca Qian VLM RALM HILM 115 20 0 11 Jul 2024
OffsetBias: Leveraging Debiased Data for Tuning Evaluators Junsoo Park Seungyeon Jwa Meiying Ren Daeyoung Kim Sanghyuk Choi ALM 87 43 0 09 Jul 2024
Evaluating Language Models for Generating and Judging Programming Feedback Charles Koutcheme Nicola Dainese Arto Hellas Sami Sarsa Juho Leinonen Syed Ashraf Paul Denny ELM 58 5 0 05 Jul 2024
Eliminating Position Bias of Language Models: A Mechanistic Approach Ziqi Wang Hanlin Zhang Xiner Li Kuan-Hao Huang Chi Han Shuiwang Ji Sham Kakade Hao Peng Heng Ji 159 20 0 01 Jul 2024
Finding Blind Spots in Evaluator LLMs with Interpretable Checklists Sumanth Doddapaneni Mohammed Safi Ur Rahman Khan Sshubam Verma Mitesh Khapra 109 16 0 19 Jun 2024
Towards Robust Evaluation: A Comprehensive Taxonomy of Datasets and Metrics for Open Domain Question Answering in the Era of Large Language Models Akchay Srivastava Atif Memon ELM 85 1 0 19 Jun 2024
Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges Aman Singh Thakur Kartik Choudhary Venkat Srinik Ramayapally Sankaran Vaidyanathan Dieuwke Hupkes ELM ALM 177 65 0 18 Jun 2024
Unveiling Implicit Table Knowledge with Question-Then-Pinpoint Reasoner for Insightful Table Summarization Kwangwook Seo Jinyoung Yeo Dongha Lee ReLM LMTD LRM 53 2 0 18 Jun 2024
Grade Score: Quantifying LLM Performance in Option Selection Dmitri Iourovitski ELM 32 1 0 17 Jun 2024
The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models Seungone Kim Juyoung Suk Ji Yong Cho Shayne Longpre Chaeeun Kim ... Sean Welleck Graham Neubig Moontae Lee Kyungjae Lee Minjoon Seo ELM ALM LM&MA 208 44 0 09 Jun 2024
Benchmark Data Contamination of Large Language Models: A Survey Cheng Xu Shuhao Guan Derek Greene Mohand-Tahar Kechadi ELM ALM 94 56 0 06 Jun 2024
SED: Self-Evaluation Decoding Enhances Large Language Models for Better Generation Ziqin Luo Haixia Han Haokun Zhao Guochao Jiang Chengyu Du Tingyun Li Jiaqing Liang Deqing Yang Yanghua Xiao 82 4 0 26 May 2024
Fennec: Fine-grained Language Model Evaluation and Correction Extended through Branching and Bridging Xiaobo Liang Haoke Zhang Helan hu Juntao Li Jun Xu Min Zhang ALM 77 3 0 20 May 2024
PHUDGE: Phi-3 as Scalable Judge Mahesh Deshwal Apoorva Chawla ALM 29 0 0 12 May 2024
Open Source Language Models Can Provide Feedback: Evaluating LLMs' Ability to Help Students Using GPT-4-As-A-Judge Charles Koutcheme Nicola Dainese Sami Sarsa Arto Hellas Juho Leinonen Paul Denny ELM ALM 77 24 0 08 May 2024
Self-Improving Customer Review Response Generation Based on LLMs Guy Azov Tatiana Pelc Adi Fledel Alon Gila Kamhi 68 2 0 06 May 2024
Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models Seungone Kim Juyoung Suk Shayne Longpre Bill Yuchen Lin Jamin Shin Sean Welleck Graham Neubig Moontae Lee Kyungjae Lee Minjoon Seo MoMe ALM ELM 147 205 0 02 May 2024
"Ask Me Anything": How Comcast Uses LLMs to Assist Agents in Real Time Scott Rome Tianwen Chen Raphael Tang Luwei Zhou Ferhan Ture 33 3 0 01 May 2024
Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models Pat Verga Sebastian Hofstatter Sophia Althammer Yixuan Su Aleksandra Piktus Arkady Arkhangorodsky Minjie Xu Naomi White Patrick Lewis ALM ELM 126 105 0 29 Apr 2024
LLM In-Context Recall is Prompt Dependent Daniel Machlab Rick Battle LLMAG 67 16 0 13 Apr 2024
Optimization-based Prompt Injection Attack to LLM-as-a-Judge Jiawen Shi Zenghui Yuan Yinuo Liu Yue Huang Pan Zhou Lichao Sun Neil Zhenqiang Gong AAML 146 57 0 26 Mar 2024
RewardBench: Evaluating Reward Models for Language Modeling Nathan Lambert Valentina Pyatkin Jacob Morrison Lester James V. Miranda Bill Yuchen Lin ... Sachin Kumar Tom Zick Yejin Choi Noah A. Smith Hanna Hajishirzi ALM 195 260 0 20 Mar 2024
Debatrix: Multi-dimensional Debate Judge with Iterative Chronological Analysis Based on LLM Jingcong Liang Rong Ye Meng Han Ruofei Lai Xinyu Zhang Xuanjing Huang Zhongyu Wei 86 8 0 12 Mar 2024
Detectors for Safe and Reliable LLMs: Implementations, Uses, and Limitations Swapnaja Achintalwar Adriana Alvarado Garcia Ateret Anaby-Tavor Ioana Baldini Sara E. Berger ... Aashka Trivedi Kush R. Varshney Dennis L. Wei Shalisha Witherspooon Marcel Zalmanovici 94 11 0 09 Mar 2024