Spurious Correlations in Reference-Free Evaluation of Text Generation

21 April 2022

Esin Durmus

Tatsunori Hashimoto

Papers citing "Spurious Correlations in Reference-Free Evaluation of Text Generation"

43 / 43 papers shown

Title
Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators Yann Dubois Balázs Galambosi Percy Liang Tatsunori Hashimoto ALM 107 382 0 06 Apr 2024
Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations Karthik Gopalakrishnan Behnam Hedayatnia Qinlang Chen Anna Gottardi Sanjeev Kwatra Anu Venkatesh Raefer Gabriel Dilek Z. Hakkani-Tür AI4MH BDL 79 331 0 23 Aug 2023
Faithful or Extractive? On Mitigating the Faithfulness-Abstractiveness Trade-off in Abstractive Summarization Faisal Ladhak Esin Durmus He He Claire Cardie Kathleen McKeown 54 65 0 31 Aug 2021
To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation Tom Kocmi C. Federmann Roman Grundkiewicz Marcin Junczys-Dowmunt Hitokazu Matsushita Arul Menezes 84 210 0 22 Jul 2021
A Comprehensive Assessment of Dialog Evaluation Metrics Yi-Ting Yeh M. Eskénazi Shikib Mehri 66 109 0 07 Jun 2021
Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics Artidoro Pagnoni Vidhisha Balachandran Yulia Tsvetkov HILM 273 310 0 27 Apr 2021
Annotating and Modeling Fine-grained Factuality in Summarization Tanya Goyal Greg Durrett HILM 58 154 0 09 Apr 2021
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics Sebastian Gehrmann Tosin Adewumi Karmanya Aggarwal Pawan Sasanka Ammanamanchi Aremu Anuoluwapo ... Nishant Subramani Wei Xu Diyi Yang Akhila Yerukola Jiawei Zhou VLM 311 285 0 02 Feb 2021
Deconstruct to Reconstruct a Configurable Evaluation Metric for Open-Domain Dialogue Systems Vitou Phy Yang Zhao Akiko Aizawa 45 55 0 01 Nov 2020
Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale Ozan Caglayan Pranava Madhyastha Lucia Specia ELM 75 36 0 26 Oct 2020
GO FIGURE: A Meta Evaluation of Factuality in Summarization Saadia Gabriel Asli Celikyilmaz Rahul Jha Yejin Choi Jianfeng Gao HILM 272 96 0 24 Oct 2020
Re-evaluating Evaluation in Text Summarization Manik Bhandari Pranav Narayan Gour A. Ashfaq Pengfei Liu Graham Neubig 147 178 0 14 Oct 2020
Evaluating Factuality in Generation with Dependency-level Entailment Tanya Goyal Greg Durrett 104 151 0 12 Oct 2020
Dialogue Response Ranking Training with Large-Scale Human Feedback Data Xiang Gao Yizhe Zhang Michel Galley Chris Brockett Bill Dolan ALM 69 105 0 15 Sep 2020
SummEval: Re-evaluating Summarization Evaluation Alexander R. Fabbri Wojciech Kry'sciñski Bryan McCann Caiming Xiong R. Socher Dragomir R. Radev HILM 97 719 0 24 Jul 2020
Evaluation of Text Generation: A Survey Asli Celikyilmaz Elizabeth Clark Jianfeng Gao ELM LM&MA 112 387 0 26 Jun 2020
Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics Nitika Mathur Tim Baldwin Trevor Cohn 52 247 0 11 Jun 2020
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization Esin Durmus He He Mona T. Diab HILM 93 395 0 07 May 2020
Learning an Unreferenced Metric for Online Dialogue Evaluation Koustuv Sinha Prasanna Parthasarathi Jasmine Wang Ryan J. Lowe William L. Hamilton Joelle Pineau OffRL 71 84 0 01 May 2020
Asking and Answering Questions to Evaluate the Factual Consistency of Summaries Alex Jinpeng Wang Kyunghyun Cho M. Lewis HILM 84 481 0 08 Apr 2020
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization Jingqing Zhang Yao-Min Zhao Mohammad Saleh Peter J. Liu RALM 3DGS 288 2,050 0 18 Dec 2019
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension M. Lewis Yinhan Liu Naman Goyal Marjan Ghazvininejad Abdel-rahman Mohamed Omer Levy Veselin Stoyanov Luke Zettlemoyer AIMat VLM 257 10,848 0 29 Oct 2019
Evaluating the Factual Consistency of Abstractive Text Summarization Wojciech Kry'sciñski Bryan McCann Caiming Xiong R. Socher HILM 106 744 0 28 Oct 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 445 20,298 0 23 Oct 2019
Fine-Tuning Language Models from Human Preferences Daniel M. Ziegler Nisan Stiennon Jeff Wu Tom B. Brown Alec Radford Dario Amodei Paul Christiano G. Irving ALM 466 1,762 0 18 Sep 2019
Wizard of Wikipedia: Knowledge-Powered Conversational agents Emily Dinan Stephen Roller Kurt Shuster Angela Fan Michael Auli Jason Weston RALM KELM 129 950 0 03 Nov 2018
BanditSum: Extractive Summarization as a Contextual Bandit Yue Dong Songlin Yang Eric Crawford H. V. Hoof Jackie C.K. Cheung 60 181 0 25 Sep 2018
Closed-Book Training to Improve Summarization Encoder Memory Yichen Jiang Joey Tianyi Zhou RALM 56 28 0 12 Sep 2018
Bottom-Up Abstractive Summarization Sebastian Gehrmann Yuntian Deng Alexander M. Rush CVBM 159 689 0 31 Aug 2018
Improving Abstraction in Text Summarization Wojciech Kry'sciñski Romain Paulus Caiming Xiong R. Socher 60 147 0 23 Aug 2018
Neural Document Summarization by Jointly Learning to Score and Select Sentences Qingyu Zhou Nan Yang Furu Wei Shaohan Huang M. Zhou Tiejun Zhao 146 322 0 06 Jul 2018
Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting Yen-Chun Chen Joey Tianyi Zhou BDL 185 584 0 28 May 2018
Soft Layer-Specific Multi-Task Summarization with Entailment and Question Generation Han Guo Ramakanth Pasunuru Joey Tianyi Zhou 56 135 0 28 May 2018
A Unified Model for Extractive and Abstractive Summarization using Inconsistency Loss W. Hsu Chieh-Kai Lin Ming-Ying Lee Kerui Min Jing Tang Min Sun CVBM 79 240 0 16 May 2018
Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies Max Grusky Mor Naaman Yoav Artzi 88 556 0 30 Apr 2018
Learning to Extract Coherent Summary via Deep Reinforcement Learning Yuxiang Wu Baotian Hu AI4TS 43 170 0 19 Apr 2018
Multi-Reward Reinforced Summarization with Saliency and Entailment Ramakanth Pasunuru Joey Tianyi Zhou 59 201 0 17 Apr 2018
Personalizing Dialogue Agents: I have a dog, do you have pets too? Saizheng Zhang Emily Dinan Jack Urbanek Arthur Szlam Douwe Kiela Jason Weston 107 1,464 0 22 Jan 2018
Faithful to the Original: Fact Aware Neural Abstractive Summarization Ziqiang Cao Furu Wei Wenjie Li Sujian Li HILM 101 371 0 13 Nov 2017
DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset Yanran Li Hui Su Xiaoyu Shen Wenjie Li Ziqiang Cao Shuzi Niu 63 1,304 0 11 Oct 2017
Get To The Point: Summarization with Pointer-Generator Networks A. See Peter J. Liu Christopher D. Manning 3DPC 301 4,019 0 14 Apr 2017
How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation Chia-Wei Liu Ryan J. Lowe Iulian Serban Michael Noseworthy Laurent Charlin Joelle Pineau 104 1,298 0 25 Mar 2016
Domain-Adversarial Training of Neural Networks Yaroslav Ganin E. Ustinova Hana Ajakan Pascal Germain Hugo Larochelle François Laviolette M. Marchand Victor Lempitsky GAN OOD 383 9,511 0 28 May 2015