RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems

11 January 2017

Chongyang Tao

Lili Mou

Dongyan Zhao

Rui Yan

ArXiv PDF HTML

Papers citing "RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems"

50 / 50 papers shown

Title
Stick to Facts: Towards Fidelity-oriented Product Description Generation Zhangming Chan Preslav Nakov Yongliang Wang Jia-Nan Li Qing Cui Kun Gai Dongyan Zhao Rui Yan 77 23 0 11 Mar 2025
BoK: Introducing Bag-of-Keywords Loss for Interpretable Dialogue Response Generation Suvodip Dey M. Desarkar OffRL 46 0 0 20 Jan 2025
Interaction Matters: An Evaluation Framework for Interactive Dialogue Assessment on English Second Language Conversations Rena Gao Carsten Roever Jey Han Lau 32 5 0 09 Jul 2024
SLIDE: A Framework Integrating Small and Large Language Models for Open-Domain Dialogues Evaluation Kun Zhao Bohao Yang Chen Tang Chenghua Lin Liang Zhan 49 5 0 24 May 2024
The Problem of Coherence in Natural Language Explanations of Recommendations Jakub Raczynski Mateusz Lango Jerzy Stefanowski 37 6 0 18 Dec 2023
ACCENT: An Automatic Event Commonsense Evaluation Metric for Open-Domain Dialogue Systems Sarik Ghazarian Yijia Shao Rujun Han Aram Galstyan Nanyun Peng 27 7 0 12 May 2023
Poor Man's Quality Estimation: Predicting Reference-Based MT Metrics Without the Reference Vilém Zouhar S. Dhuliawala Wangchunshu Zhou Nico Daheim Tom Kocmi Yuchen Eleanor Jiang Mrinmaya Sachan 18 9 0 21 Jan 2023
Follow the Timeline! Generating Abstractive and Extractive Timeline Summary in Chronological Order Preslav Nakov Li Mingzhe Shen Gao Zhangming Chan Dongyan Zhao Xin Gao Xiangliang Zhang Rui Yan 157 4 0 02 Jan 2023
MAUVE Scores for Generative Models: Theory and Practice Krishna Pillutla Lang Liu John Thickstun Sean Welleck Swabha Swayamdipta Rowan Zellers Sewoong Oh Yejin Choi Zaïd Harchaoui EGVM 44 22 0 30 Dec 2022
MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation Jiazhan Feng Qingfeng Sun Can Xu Pu Zhao Yaming Yang Chongyang Tao Dongyan Zhao Qingwei Lin 32 52 0 10 Nov 2022
EnDex: Evaluation of Dialogue Engagingness at Scale Guangxuan Xu Ruibo Liu Fabrice Harel-Canada Nischal Reddy Chandra Nanyun Peng 21 5 0 22 Oct 2022
StoryER: Automatic Story Evaluation via Ranking, Rating and Reasoning Hong Chen D. Vo Hiroya Takamura Yusuke Miyao Hideki Nakayama 30 20 0 16 Oct 2022
SelF-Eval: Self-supervised Fine-grained Dialogue Evaluation Longxuan Ma Ziyu Zhuang Weinan Zhang Mingda Li Ting Liu 29 4 0 17 Aug 2022
MME-CRS: Multi-Metric Evaluation Based on Correlation Re-Scaling for Evaluating Open-Domain Dialogue Pengfei Zhang Xiao-fei Hu Kaidong Yu Jian Wang Song-Bo Han Cao Liu C. Yuan 27 7 0 19 Jun 2022
Target-Guided Dialogue Response Generation Using Commonsense and Data Augmentation Prakhar Gupta Harsh Jhamtani Jeffrey P. Bigham 49 12 0 19 May 2022
CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation Pei Ke Hao Zhou Yankai Lin Peng Li Jie Zhou Xiaoyan Zhu Minlie Huang 21 38 0 02 Apr 2022
What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation Sarik Ghazarian Behnam Hedayatnia Alexandros Papangelis Yang Liu Dilek Z. Hakkani-Tür 30 19 0 25 Mar 2022
Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challenges Shikib Mehri Jinho Choi L. F. D’Haro Jan Deriu M. Eskénazi ... David Traum Yi-Ting Yeh Zhou Yu Yizhe Zhang Chen Zhang 34 21 0 18 Mar 2022
DEAM: Dialogue Coherence Evaluation using AMR-based Semantic Manipulations Sarik Ghazarian Nuan Wen Aram Galstyan Nanyun Peng 27 40 0 18 Mar 2022
Probing the Robustness of Trained Metrics for Conversational Dialogue Systems Jan Deriu Don Tuggener Pius von Daniken Mark Cieliebak AAML 19 9 0 28 Feb 2022
Rethinking and Refining the Distinct Metric Siyang Liu Sahand Sabour Yinhe Zheng Pei Ke Xiaoyan Zhu Minlie Huang 36 11 0 28 Feb 2022
FlowEval: A Consensus-Based Dialogue Evaluation Framework Using Segment Act Flows Jianqiao Zhao Yanyang Li Wanyu Du Yangfeng Ji Dong Yu M. Lyu Liwei Wang 33 4 0 14 Feb 2022
Ditch the Gold Standard: Re-evaluating Conversational Question Answering Huihan Li Tianyu Gao Manan Goenka Danqi Chen 24 21 0 16 Dec 2021
MDD-Eval: Self-Training on Augmented Data for Multi-Domain Dialogue Evaluation Chen Zhang L. F. D’Haro Thomas Friedrichs Haizhou Li ELM 25 18 0 14 Dec 2021
Identifying Untrustworthy Samples: Data Filtering for Open-domain Dialogues with Bayesian Optimization Lei Shen Haolan Zhan Xin Shen Hongshen Chen Xiaofang Zhao Xiao-Dan Zhu 43 17 0 14 Sep 2021
Perturbation CheckLists for Evaluating NLG Evaluation Metrics Ananya B. Sai Tanay Dixit D. Y. Sheth S. Mohan Mitesh M. Khapra AAML 116 58 0 13 Sep 2021
How to Evaluate Your Dialogue Models: A Review of Approaches Xinmeng Li Wansen Wu Long Qin Quanjun Yin ELM 30 8 0 03 Aug 2021
WeaSuL: Weakly Supervised Dialogue Policy Learning: Reward Estimation for Multi-turn Dialogue Anant Khandelwal OffRL 24 6 0 01 Aug 2021
Synthesizing Adversarial Negative Responses for Robust Response Ranking and Evaluation Prakhar Gupta Yulia Tsvetkov Jeffrey P. Bigham 42 22 0 10 Jun 2021
A Comprehensive Assessment of Dialog Evaluation Metrics Yi-Ting Yeh M. Eskénazi Shikib Mehri 36 105 0 07 Jun 2021
GTM: A Generative Triple-Wise Model for Conversational Question Generation Lei Shen Fandong Meng Jinchao Zhang Yang Feng Jie Zhou 19 13 0 07 Jun 2021
DynaEval: Unifying Turn and Dialogue Level Evaluation Chen Zhang Yiming Chen L. F. D’Haro Yan Zhang Thomas Friedrichs Grandee Lee Haizhou Li 24 73 0 02 Jun 2021
HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversations Weixin Liang Kai-Hui Liang Zhou Yu 42 15 0 01 Jun 2021
OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metrics Jian Guan Zhexin Zhang Zhuoer Feng Zitao Liu Wenbiao Ding Xiaoxi Mao Changjie Fan Minlie Huang 20 60 0 19 May 2021
Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey Jinjie Ni Tom Young Vlad Pandelea Fuzhao Xue Min Zhang 54 268 0 10 May 2021
Meta-evaluation of Conversational Search Evaluation Metrics Zeyang Liu K. Zhou Max L. Wilson ELM 32 17 0 27 Apr 2021
CLIPScore: A Reference-free Evaluation Metric for Image Captioning Jack Hessel Ari Holtzman Maxwell Forbes Ronan Le Bras Yejin Choi CLIP 17 1,448 0 18 Apr 2021
MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers Krishna Pillutla Swabha Swayamdipta Rowan Zellers John Thickstun Sean Welleck Yejin Choi Zaïd Harchaoui 45 343 0 02 Feb 2021
Deconstruct to Reconstruct a Configurable Evaluation Metric for Open-Domain Dialogue Systems Vitou Phy Yang Zhao Akiko Aizawa 14 55 0 01 Nov 2020
UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation Jian Guan Minlie Huang 29 69 0 16 Sep 2020
A Survey of Evaluation Metrics Used for NLG Systems Ananya B. Sai Akash Kumar Mohankumar Mitesh M. Khapra ELM 33 230 0 27 Aug 2020
Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluation Weixin Liang James Zou Zhou Yu ELM 34 33 0 21 May 2020
Learning an Unreferenced Metric for Online Dialogue Evaluation Koustuv Sinha Prasanna Parthasarathi Jasmine Wang Ryan J. Lowe William L. Hamilton Joelle Pineau OffRL 29 84 0 01 May 2020
A Survey of Document Grounded Dialogue Systems (DGDS) Longxuan Ma Weinan Zhang Mingda Li Ting Liu 32 19 0 17 Apr 2020
Towards a Human-like Open-Domain Chatbot Daniel De Freitas Minh-Thang Luong David R. So Jamie Hall Noah Fiedel ... Zi Yang Apoorv Kulshreshtha Gaurav Nemade Yifeng Lu Quoc V. Le 42 924 0 27 Jan 2020
Deep Learning Based Chatbot Models Richard Csaky 29 46 0 23 Aug 2019
A Multi-Turn Emotionally Engaging Dialog Model Yubo Xie Ekaterina Svikhnushina P. Pu 16 15 0 15 Aug 2019
Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators Sanghyun Yi Rahul Goel Chandra Khatri Alessandra Cervone Tagyoung Chung Behnam Hedayatnia Anu Venkatesh Raefer Gabriel Dilek Z. Hakkani-Tür 20 60 0 30 Apr 2019
Response Generation by Context-aware Prototype Editing Yu Wu Furu Wei Shaohan Huang Yunli Wang Zhoujun Li Ming Zhou KELM 21 121 0 19 Jun 2018
A Survey on Dialogue Systems: Recent Advances and New Frontiers Hongshen Chen Xiaorui Liu Dawei Yin Jiliang Tang VLM LLMAG 38 695 0 06 Nov 2017