BUMP: A Benchmark of Unfaithful Minimal Pairs for Meta-Evaluation of Faithfulness Metrics

20 December 2022

Papers citing "BUMP: A Benchmark of Unfaithful Minimal Pairs for Meta-Evaluation of Faithfulness Metrics"

24 / 24 papers shown

Title
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization Alexander R. Fabbri Chien-Sheng Wu Wenhao Liu Caiming Xiong HILM 53 216 0 16 Dec 2021
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization Philippe Laban Tobias Schnabel Paul N. Bennett Marti A. Hearst HILM 83 390 0 18 Nov 2021
Learning Compact Metrics for MT Amy Pu Hyung Won Chung Ankur P. Parikh Sebastian Gehrmann Thibault Sellam 59 100 0 12 Oct 2021
Factual Consistency Evaluation for Text Summarization via Counterfactual Estimation Yuexiang Xie Fei Sun Yang Deng Yaliang Li Bolin Ding HILM 47 53 0 30 Aug 2021
BARTScore: Evaluating Generated Text as Text Generation Weizhe Yuan Graham Neubig Pengfei Liu 93 838 0 22 Jun 2021
Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics Artidoro Pagnoni Vidhisha Balachandran Yulia Tsvetkov HILM 253 310 0 27 Apr 2021
$$Q^{2}$: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering$ $Q^{2}$ : Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering Or Honovich Leshem Choshen Roee Aharoni Ella Neeman Idan Szpektor Omri Abend HILM 57 139 0 16 Apr 2021
QuestEval: Summarization Asks for Fact-based Evaluation Thomas Scialom Paul-Alexis Dray Patrick Gallinari Sylvain Lamprier Benjamin Piwowarski Jacopo Staiano Alex Jinpeng Wang HILM 48 274 0 23 Mar 2021
GO FIGURE: A Meta Evaluation of Factuality in Summarization Saadia Gabriel Asli Celikyilmaz Rahul Jha Yejin Choi Jianfeng Gao HILM 255 96 0 24 Oct 2020
Evaluating Factuality in Generation with Dependency-level Entailment Tanya Goyal Greg Durrett 90 150 0 12 Oct 2020
SummEval: Re-evaluating Summarization Evaluation Alexander R. Fabbri Wojciech Kry'sciñski Bryan McCann Caiming Xiong R. Socher Dragomir R. Radev HILM 90 709 0 24 Jul 2020
DeBERTa: Decoding-enhanced BERT with Disentangled Attention Pengcheng He Xiaodong Liu Jianfeng Gao Weizhu Chen AAML 118 2,720 0 05 Jun 2020
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization Esin Durmus He He Mona T. Diab HILM 77 392 0 07 May 2020
On Faithfulness and Factuality in Abstractive Summarization Joshua Maynez Shashi Narayan Bernd Bohnet Ryan T. McDonald HILM 75 1,028 0 02 May 2020
BLEURT: Learning Robust Metrics for Text Generation Thibault Sellam Dipanjan Das Ankur P. Parikh 77 1,489 0 09 Apr 2020
Asking and Answering Questions to Evaluate the Factual Consistency of Summaries Alex Jinpeng Wang Kyunghyun Cho M. Lewis HILM 71 479 0 08 Apr 2020
BLiMP: The Benchmark of Linguistic Minimal Pairs for English Alex Warstadt Alicia Parrish Haokun Liu Anhad Mohananey Wei Peng Sheng-Fu Wang Samuel R. Bowman 72 491 0 02 Dec 2019
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension M. Lewis Yinhan Liu Naman Goyal Marjan Ghazvininejad Abdel-rahman Mohamed Omer Levy Veselin Stoyanov Luke Zettlemoyer AIMat VLM 190 10,787 0 29 Oct 2019
Evaluating the Factual Consistency of Abstractive Text Summarization Wojciech Kry'sciñski Bryan McCann Caiming Xiong R. Socher HILM 85 742 0 28 Oct 2019
Neural Text Summarization: A Critical Evaluation Wojciech Kry'sciñski N. Keskar Bryan McCann Caiming Xiong R. Socher 72 365 0 23 Aug 2019
BERTScore: Evaluating Text Generation with BERT Tianyi Zhang Varsha Kishore Felix Wu Kilian Q. Weinberger Yoav Artzi 256 5,764 0 21 Apr 2019
Targeted Syntactic Evaluation of Language Models Rebecca Marvin Tal Linzen 70 415 0 27 Aug 2018
Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization Shashi Narayan Shay B. Cohen Mirella Lapata AILaw 111 1,670 0 27 Aug 2018
Teaching Machines to Read and Comprehend Karl Moritz Hermann Tomás Kociský Edward Grefenstette L. Espeholt W. Kay Mustafa Suleyman Phil Blunsom 304 3,538 0 10 Jun 2015