Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics

27 April 2021

Artidoro Pagnoni

Vidhisha Balachandran

Yulia Tsvetkov

HILM

ArXiv PDF HTML

Papers citing "Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics"

50 / 64 papers shown

Title
Statistical Deficiency for Task Inclusion Estimation Loïc Fosse Frédéric Béchet Benoit Favre Géraldine Damnati Gwénolé Lecorvé Maxime Darrin Philippe Formont Pablo Piantanida 133 0 0 07 Mar 2025
Learning to Summarize from LLM-generated Feedback Hwanjun Song Taewon Yun Yuho Lee Jihwan Oh Gihun Lee Jason (Jinglun) Cai Hang Su 73 2 0 28 Jan 2025
Improving Model Factuality with Fine-grained Critique-based Evaluator Yiqing Xie Wenxuan Zhou Pradyot Prakash Di Jin Yuning Mao ... Sinong Wang Han Fang Carolyn Rose Daniel Fried Hejia Zhang HILM 33 5 0 24 Oct 2024
Do Robot Snakes Dream like Electric Sheep? Investigating the Effects of Architectural Inductive Biases on Hallucination Jerry Huang Prasanna Parthasarathi Mehdi Rezagholizadeh Boxing Chen Sarath Chandar 48 0 0 22 Oct 2024
CREAM: Comparison-Based Reference-Free ELO-Ranked Automatic Evaluation for Meeting Summarization Ziwei Gong Lin Ai Harshsaiprasad Deshpande Alexander Johnson Emmy Phung Zehui Wu Ahmad Emami Julia Hirschberg 36 2 0 17 Sep 2024
STORYSUMM: Evaluating Faithfulness in Story Summarization Melanie Subbiah Faisal Ladhak Akankshya Mishra Griffin Adams Lydia B. Chilton Kathleen McKeown 34 4 0 09 Jul 2024
FineSurE: Fine-grained Summarization Evaluation using LLMs Hwanjun Song Hang Su Igor Shalyminov Jason (Jinglun) Cai Saab Mansour HILM 28 29 0 01 Jul 2024
Language in Vivo vs. in Silico: Size Matters but Larger Language Models Still Do Not Comprehend Language on a Par with Humans Vittoria Dentella Fritz Guenther Evelina Leivada ELM 41 1 0 23 Apr 2024
A Closer Look at Claim Decomposition Miriam Wanner Seth Ebner Zhengping Jiang Mark Dredze Benjamin Van Durme 47 18 0 18 Mar 2024
Evaluating LLMs for Gender Disparities in Notable Persons L. Rhue Sofie Goethals Arun Sundararajan 41 4 0 14 Mar 2024
Fine-Grained Natural Language Inference Based Faithfulness Evaluation for Diverse Summarisation Tasks Huajian Zhang Yumo Xu Laura Perez-Beltrachini HILM 24 9 0 27 Feb 2024
Does the Generator Mind its Contexts? An Analysis of Generative Model Faithfulness under Context Transfer Xinshuo Hu Baotian Hu Dongfang Li Xiaoguang Li Lifeng Shang HILM 23 1 0 22 Feb 2024
EHRTutor: Enhancing Patient Understanding of Discharge Instructions Zihao Zhang Zonghai Yao Huixue Zhou Feiyun Ouyang Hong-ye Yu LM&MA AI4Ed 30 4 0 30 Oct 2023
KGQuiz: Evaluating the Generalization of Encoded Knowledge in Large Language Models Yuyang Bai Shangbin Feng Vidhisha Balachandran Zhaoxuan Tan Shiqi Lou Tianxing He Yulia Tsvetkov ELM 40 2 0 15 Oct 2023
Thresh: A Unified, Customizable and Deployable Platform for Fine-Grained Text Evaluation David Heineman Yao Dou Wei-ping Xu 22 7 0 14 Aug 2023
Named Entity Inclusion in Abstractive Text Summarization S. Berezin Tatiana Batura 28 7 0 05 Jul 2023
Annotating and Detecting Fine-grained Factual Errors for Dialogue Summarization Rongxin Zhu Jianzhong Qi Jey Han Lau 28 9 0 26 May 2023
MuLER: Detailed and Scalable Reference-based Evaluation Taelin Karidi Leshem Choshen Gal Patel Omri Abend 25 0 0 24 May 2023
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation Sewon Min Kalpesh Krishna Xinxi Lyu M. Lewis Wen-tau Yih Pang Wei Koh Mohit Iyyer Luke Zettlemoyer Hannaneh Hajishirzi HILM ALM 54 599 0 23 May 2023
Evaluating Factual Consistency of Summaries with Large Language Models Shiqi Chen Siyang Gao Junxian He ELM LRM HILM 22 6 0 23 May 2023
APPLS: Evaluating Evaluation Metrics for Plain Language Summarization Yue Guo Tal August Gondy Leroy T. Cohen Lucy Lu Wang 55 9 0 23 May 2023
Evaluating Factual Consistency of Texts with Semantic Role Labeling Jing Fan Dennis Aumiller Michael Gertz HILM 15 4 0 22 May 2023
What are the Desired Characteristics of Calibration Sets? Identifying Correlates on Long Form Scientific Summarization Griffin Adams Bichlien H. Nguyen Jake A. Smith Yingce Xia Shufang Xie Anna Ostropolets Budhaditya Deb Yuan Chen Tristan Naumann Noémie Elhadad 22 8 0 12 May 2023
The Current State of Summarization Fabian Retkowski 23 6 0 08 May 2023
Automatically Summarizing Evidence from Clinical Trials: A Prototype Highlighting Current Challenges S. Ramprasad Denis Jered McInerney Iain J. Marshal Byron C. Wallace 18 9 0 07 Mar 2023
Contrastive Error Attribution for Finetuned Language Models Faisal Ladhak Esin Durmus Tatsunori Hashimoto HILM 25 9 0 21 Dec 2022
WeCheck: Strong Factual Consistency Checker via Weakly Supervised Learning Wenhao Wu Wei Li Xinyan Xiao Jiachen Liu Sujian Li Yajuan Lv HILM 23 4 0 20 Dec 2022
On the Blind Spots of Model-Based Evaluation Metrics for Text Generation Tianxing He Jingyu Zhang Tianle Wang Sachin Kumar Kyunghyun Cho James R. Glass Yulia Tsvetkov 37 44 0 20 Dec 2022
On Improving Summarization Factual Consistency from Natural Language Feedback Yixin Liu Budhaditya Deb Milagro Teruel Aaron L Halfaker Dragomir R. Radev Ahmed Hassan Awadallah HILM 19 35 0 20 Dec 2022
BUMP: A Benchmark of Unfaithful Minimal Pairs for Meta-Evaluation of Faithfulness Metrics Liang Ma Shuyang Cao IV RobertL.Logan Di Lu Shihao Ran Kecheng Zhang Joel R. Tetreault A. Jaimes 13 6 0 20 Dec 2022
Evaluating Human-Language Model Interaction Mina Lee Megha Srivastava Amelia Hardy John Thickstun Esin Durmus ... Hancheng Cao Tony Lee Rishi Bommasani Michael S. Bernstein Percy Liang LM&MA ALM 46 99 0 19 Dec 2022
SumREN: Summarizing Reported Speech about Events in News R. Reddy Heba Elfardy Hou Pong Chan Kevin Small Heng Ji 24 5 0 02 Dec 2022
GoSum: Extractive Summarization of Long Documents by Reinforcement Learning and Graph Organized discourse state Junyi Bian Xiaodi Huang Hong Zhou Shanfeng Zhu 19 11 0 18 Nov 2022
Improving Factual Consistency in Summarization with Compression-Based Post-Editing Alexander R. Fabbri Prafulla Kumar Choubey Jesse Vig Chien-Sheng Wu Caiming Xiong HILM KELM 42 17 0 11 Nov 2022
Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learning Yu Meng Martin Michalski Jiaxin Huang Yu Zhang Tarek F. Abdelzaher Jiawei Han VLM 43 46 0 06 Nov 2022
FRSUM: Towards Faithful Abstractive Summarization via Enhancing Factual Robustness Wenhao Wu Wei Li Jiachen Liu Xinyan Xiao Ziqiang Cao Sujian Li Hua-Hong Wu HILM 19 10 0 01 Nov 2022
SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control Xiaochuang Han Sachin Kumar Yulia Tsvetkov 32 79 0 31 Oct 2022
How Far are We from Robust Long Abstractive Summarization? Huan Yee Koh Jiaxin Ju He Zhang Ming Liu Shirui Pan HILM 23 39 0 30 Oct 2022
Evaluating Parameter Efficient Learning for Generation Peng-Tao Xu M. Patwary Shrimai Prabhumoye Virginia Adams R. Prenger Wei Ping Nayeon Lee M. Shoeybi Bryan Catanzaro MoE 20 3 0 25 Oct 2022
Mutual Information Alleviates Hallucinations in Abstractive Summarization Liam van der Poel Ryan Cotterell Clara Meister HILM 11 56 0 24 Oct 2022
Precisely the Point: Adversarial Augmentations for Faithful and Informative Text Generation Wenhao Wu Wei Li Jiachen Liu Xinyan Xiao Sujian Li Yajuan Lyu 31 3 0 22 Oct 2022
Model Criticism for Long-Form Text Generation Yuntian Deng Volodymyr Kuleshov Alexander M. Rush 31 19 0 16 Oct 2022
Just ClozE! A Novel Framework for Evaluating the Factual Consistency Faster in Abstractive Summarization Yiyang Li Lei Li Marina Litvak N. Vanetik Dingxing Hu Yuze Li Yanquan Zhou HILM 30 0 0 06 Oct 2022
An Empirical Survey on Long Document Summarization: Datasets, Models and Metrics Huan Yee Koh Jiaxin Ju Ming Liu Shirui Pan 73 122 0 03 Jul 2022
Gradient-Based Constrained Sampling from Language Models Sachin Kumar Biswajit Paria Yulia Tsvetkov BDL 30 53 0 25 May 2022
SQuALITY: Building a Long-Document Summarization Dataset the Hard Way Alex Jinpeng Wang Richard Yuanzhe Pang Angelica Chen Jason Phang Samuel R. Bowman 74 44 0 23 May 2022
Near-Negative Distinction: Giving a Second Life to Human Evaluation Datasets Philippe Laban Chien-Sheng Wu Wenhao Liu Caiming Xiong 33 5 0 13 May 2022
Masked Summarization to Generate Factually Inconsistent Summaries for Improved Factual Consistency Checking Hwanhee Lee Kang Min Yoo Joonsuk Park Hwaran Lee Kyomin Jung HILM 13 10 0 04 May 2022
QRelScore: Better Evaluating Generated Questions with Deeper Understanding of Context-aware Relevance Xiaoqiang Wang Bang Liu Siliang Tang Lingfei Wu 16 9 0 29 Apr 2022
Faithful to the Document or to the World? Mitigating Hallucinations via Entity-linked Knowledge in Abstractive Summarization Yue Dong John Wieting Pat Verga HILM 16 24 0 28 Apr 2022