Leveraging Large Language Models for NLG Evaluation: Advances and Challenges

13 January 2024

Papers citing "Leveraging Large Language Models for NLG Evaluation: Advances and Challenges"

39 / 39 papers shown

Title
LLM-as-a-Judge: Reassessing the Performance of LLMs in Extractive QA Xanh Ho Jiahao Huang Florian Boudin Akiko Aizawa ELM 79 0 0 16 Apr 2025
PRD: Peer Rank and Discussion Improve Large Language Model based Evaluations Ruosen Li Teerth Patel Xinya Du LLMAG ALM 124 101 0 03 Jan 2025
RevisEval: Improving LLM-as-a-Judge via Response-Adapted References Qiyuan Zhang Yufei Wang Tiezheng YU Yuxin Jiang Chuhan Wu ... Xin Jiang Lifeng Shang Ruiming Tang Fuyuan Lyu Chen Ma 66 6 0 07 Oct 2024
Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models Seungone Kim Juyoung Suk Shayne Longpre Bill Yuchen Lin Jamin Shin Sean Welleck Graham Neubig Moontae Lee Kyungjae Lee Minjoon Seo MoMe ALM ELM 89 195 0 02 May 2024
On the Challenges and Opportunities in Generative AI Laura Manduchi Kushagra Pandey Robert Bamler Ryan Cotterell Sina Daubener ... F. Wenzel Frank Wood Stephan Mandt Vincent Fortuin Vincent Fortuin 181 21 0 28 Feb 2024
Towards Better Evaluation of Instruction-Following: A Case-Study in Summarization Ondrej Skopek Rahul Aralikatte Sian Gooding Victor Carbune ELM 67 19 0 12 Oct 2023
TIGERScore: Towards Building Explainable Metric for All Text Generation Tasks Dongfu Jiang Yishan Li Ge Zhang Wenhao Huang Bill Yuchen Lin Wenhu Chen ALM 67 66 0 01 Oct 2023
Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations Karthik Gopalakrishnan Behnam Hedayatnia Qinlang Chen Anna Gottardi Sanjeev Kwatra Anu Venkatesh Raefer Gabriel Dilek Z. Hakkani-Tür AI4MH BDL 67 331 0 23 Aug 2023
Wider and Deeper LLM Networks are Fairer LLM Evaluators Xinghua Zhang Yu Bowen Haiyang Yu Yangyu Lv Tingwen Liu Fei Huang Hongbo Xu Yongbin Li ALM 102 86 0 03 Aug 2023
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena Lianmin Zheng Wei-Lin Chiang Ying Sheng Siyuan Zhuang Zhanghao Wu ... Dacheng Li Eric Xing Haotong Zhang Joseph E. Gonzalez Ion Stoica ALM OSLM ELM 312 4,253 0 09 Jun 2023
UMSE: Unified Multi-scenario Summarization Evaluation Shen Gao Zhitao Yao Chongyang Tao Preslav Nakov Pengjie Ren Zhaochun Ren Zhumin Chen 65 5 0 26 May 2023
Investigating Table-to-Text Generation Capabilities of LLMs in Real-World Information Seeking Scenarios Yilun Zhao Haowei Zhang Shengyun Si Linyong Nan Xiangru Tang Arman Cohan LMTD 62 12 0 24 May 2023
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation Sewon Min Kalpesh Krishna Xinxi Lyu M. Lewis Wen-tau Yih Pang Wei Koh Mohit Iyyer Luke Zettlemoyer Hannaneh Hajishirzi HILM ALM 113 678 0 23 May 2023
Stylized Data-to-Text Generation: A Case Study in the E-Commerce Domain Liqiang Jing Xuemeng Song Xuming Lin Zhongzhou Zhao Wei Zhou Liqiang Nie 80 15 0 05 May 2023
Large Language Models are Diverse Role-Players for Summarization Evaluation Ning Wu Ming Gong Linjun Shou Shining Liang Daxin Jiang 111 67 0 27 Mar 2023
Is ChatGPT a Good NLG Evaluator? A Preliminary Study Jiaan Wang Yunlong Liang Fandong Meng Zengkui Sun Haoxiang Shi Zhixu Li Jinan Xu Jianfeng Qu Jie Zhou LM&MA ELM ALM AI4MH 103 464 0 07 Mar 2023
Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech Fan Huang Haewoon Kwak Jisun An AI4MH 64 264 0 11 Feb 2023
T5Score: Discriminative Fine-tuning of Generative Evaluation Metrics Yiwei Qin Weizhe Yuan Graham Neubig Pengfei Liu 41 23 0 12 Dec 2022
Scaling Instruction-Finetuned Language Models Hyung Won Chung Le Hou Shayne Longpre Barret Zoph Yi Tay ... Jacob Devlin Adam Roberts Denny Zhou Quoc V. Le Jason W. Wei ReLM LRM 167 3,110 0 20 Oct 2022
ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic Yoad Tewel Yoav Shalev Idan Schwartz Lior Wolf VLM 52 194 0 29 Nov 2021
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization Philippe Laban Tobias Schnabel Paul N. Bennett Marti A. Hearst HILM 90 390 0 18 Nov 2021
Finetuned Language Models Are Zero-Shot Learners Jason W. Wei Maarten Bosma Vincent Zhao Kelvin Guu Adams Wei Yu Brian Lester Nan Du Andrew M. Dai Quoc V. Le ALM UQCV 116 3,723 0 03 Sep 2021
BARTScore: Evaluating Generated Text as Text Generation Weizhe Yuan Graham Neubig Pengfei Liu 95 838 0 22 Jun 2021
OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metrics Jian Guan Zhexin Zhang Zhuoer Feng Zitao Liu Wenbiao Ding Xiaoxi Mao Changjie Fan Minlie Huang 54 61 0 19 May 2021
Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics Artidoro Pagnoni Vidhisha Balachandran Yulia Tsvetkov HILM 260 310 0 27 Apr 2021
Re-evaluating Evaluation in Text Summarization Manik Bhandari Pranav Narayan Gour A. Ashfaq Pengfei Liu Graham Neubig 118 176 0 14 Oct 2020
Evaluation of Text Generation: A Survey Asli Celikyilmaz Elizabeth Clark Jianfeng Gao ELM LM&MA 94 384 0 26 Jun 2020
Unsupervised Evaluation of Interactive Dialog with DialoGPT Shikib Mehri M. Eskénazi 54 177 0 23 Jun 2020
USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation Shikib Mehri M. Eskénazi 46 223 0 01 May 2020
Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing Brian Thompson Matt Post LRM 53 190 0 30 Apr 2020
Asking and Answering Questions to Evaluate the Factual Consistency of Summaries Alex Jinpeng Wang Kyunghyun Cho M. Lewis HILM 77 479 0 08 Apr 2020
Towards a Human-like Open-Domain Chatbot Daniel De Freitas Minh-Thang Luong David R. So Jamie Hall Noah Fiedel ... Zi Yang Apoorv Kulshreshtha Gaurav Nemade Yifeng Lu Quoc V. Le 91 935 0 27 Jan 2020
SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization Bogdan Gliwa Iwona Mochol M. Biesek A. Wawer 111 631 0 27 Nov 2019
Simple, Scalable Adaptation for Neural Machine Translation Ankur Bapna N. Arivazhagan Orhan Firat AI4CE 92 416 0 18 Sep 2019
Learning to Evaluate Image Captioning Huayu Chen Guandao Yang Andreas Veit Xun Huang Serge J. Belongie 62 148 0 17 Jun 2018
Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies Max Grusky Mor Naaman Yoav Artzi 84 552 0 30 Apr 2018
Personalizing Dialogue Agents: I have a dog, do you have pets too? Saizheng Zhang Emily Dinan Jack Urbanek Arthur Szlam Douwe Kiela Jason Weston 89 1,453 0 22 Jan 2018
Teaching Machines to Read and Comprehend Karl Moritz Hermann Tomás Kociský Edward Grefenstette L. Espeholt W. Kay Mustafa Suleyman Phil Blunsom 319 3,538 0 10 Jun 2015
CIDEr: Consensus-based Image Description Evaluation Ramakrishna Vedantam C. L. Zitnick Devi Parikh 252 4,471 0 20 Nov 2014