Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses

23 August 2017

Ryan J. Lowe

Michael Noseworthy

Iulian Serban

Nicolas Angelard-Gontier

Yoshua Bengio

Joelle Pineau

ArXiv PDF HTML

Papers citing "Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses"

50 / 79 papers shown

Title
BoK: Introducing Bag-of-Keywords Loss for Interpretable Dialogue Response Generation Suvodip Dey M. Desarkar OffRL 46 0 0 20 Jan 2025
Measuring the Robustness of Reference-Free Dialogue Evaluation Systems Justin Vasselli Adam Nohejl Taro Watanabe AAML 54 0 0 12 Jan 2025
Apollonion: Profile-centric Dialog Agent Shangyu Chen Zibo Zhao Yuanyuan Zhao Xiang Li LLMAG 40 1 0 10 Apr 2024
Three Ways of Using Large Language Models to Evaluate Chat Ondvrej Plátek Vojtvech Hudevcek Patrícia Schmidtová Mateusz Lango Ondrej Dusek ALM 19 6 0 12 Aug 2023
The Turing Deception David Noever Matt Ciolino DeLMO ELM LRM 80 9 0 09 Dec 2022
Open-world Story Generation with Structured Knowledge Enhancement: A Comprehensive Survey Yuxin Wang Jieru Lin Zhiwei Yu Wei Hu Börje F. Karlsson 20 17 0 09 Dec 2022
Bipartite-play Dialogue Collection for Practical Automatic Evaluation of Dialogue Systems Shiki Sato Yosuke Kishinami Hiroaki Sugiyama Reina Akama Ryoko Tokuhisa Jun Suzuki 21 2 0 19 Nov 2022
Generative Transformers for Design Concept Generation Qihao Zhu Jianxi Luo AI4CE 25 48 0 07 Nov 2022
Deepfake Text Detection: Limitations and Opportunities Jiameng Pu Zain Sarwar Sifat Muhammad Abdullah A. Rehman Yoonjin Kim P. Bhattacharya M. Javed Bimal Viswanath AAML 24 54 0 17 Oct 2022
StoryER: Automatic Story Evaluation via Ranking, Rating and Reasoning Hong Chen D. Vo Hiroya Takamura Yusuke Miyao Hideki Nakayama 27 20 0 16 Oct 2022
SelF-Eval: Self-supervised Fine-grained Dialogue Evaluation Longxuan Ma Ziyu Zhuang Weinan Zhang Mingda Li Ting Liu 29 4 0 17 Aug 2022
A Comprehensive Survey of Natural Language Generation Advances from the Perspective of Digital Deception Keenan I. Jones Enes ALTUNCU V. N. Franqueira Yi-Chia Wang Shujun Li DeLMO 39 3 0 11 Aug 2022
CORAL: Contextual Response Retrievability Loss Function for Training Dialog Generation Models Bishal Santra Ravi Ghadia Manish Gupta Pawan Goyal OffRL 23 0 0 21 May 2022
Target-Guided Dialogue Response Generation Using Commonsense and Data Augmentation Prakhar Gupta Harsh Jhamtani Jeffrey P. Bigham 49 12 0 19 May 2022
Empathetic Response Generation with State Management Yuhan Liu Jun Gao Jiachen Du Lanjun Zhou Ruifeng Xu 29 0 0 07 May 2022
Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generation Kushal Arora Layla El Asri Hareesh Bahuleyan Jackie C.K. Cheung 40 79 0 03 Apr 2022
What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation Sarik Ghazarian Behnam Hedayatnia Alexandros Papangelis Yang Liu Dilek Z. Hakkani-Tür 30 19 0 25 Mar 2022
Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challenges Shikib Mehri Jinho Choi L. F. D’Haro Jan Deriu M. Eskénazi ... David Traum Yi-Ting Yeh Zhou Yu Yizhe Zhang Chen Zhang 30 21 0 18 Mar 2022
DEAM: Dialogue Coherence Evaluation using AMR-based Semantic Manipulations Sarik Ghazarian Nuan Wen Aram Galstyan Nanyun Peng 27 40 0 18 Mar 2022
RoMe: A Robust Metric for Evaluating Natural Language Generation Md. Rony Liubov Kovriguina Debanjan Chaudhuri Ricardo Usbeck Jens Lehmann 22 12 0 17 Mar 2022
Probing the Robustness of Trained Metrics for Conversational Dialogue Systems Jan Deriu Don Tuggener Pius von Daniken Mark Cieliebak AAML 19 9 0 28 Feb 2022
A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models Hanqing Zhang Haolin Song Shaoyu Li Ming Zhou Dawei Song 52 214 0 14 Jan 2022
A Survey of Natural Language Generation Chenhe Dong Hai-Tao Zheng Haifan Gong Mengzhao Chen Junxin Li Ying Shen Min Yang 3DV 27 43 0 22 Dec 2021
Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation Mingkai Deng Bowen Tan Zhengzhong Liu Eric Xing Zhiting Hu 16 72 0 14 Sep 2021
Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions Mohammad Aliannejadi Julia Kiseleva A. Chuklin Jeffrey Stephen Dalton Andrey Kravchenko 79 97 0 13 Sep 2021
Perturbation CheckLists for Evaluating NLG Evaluation Metrics Ananya B. Sai Tanay Dixit D. Y. Sheth S. Mohan Mitesh M. Khapra AAML 116 57 0 13 Sep 2021
How to Evaluate Your Dialogue Models: A Review of Approaches Xinmeng Li Wansen Wu Long Qin Quanjun Yin ELM 30 8 0 03 Aug 2021
WeaSuL: Weakly Supervised Dialogue Policy Learning: Reward Estimation for Multi-turn Dialogue Anant Khandelwal OffRL 24 6 0 01 Aug 2021
Synthesizing Adversarial Negative Responses for Robust Response Ranking and Evaluation Prakhar Gupta Yulia Tsvetkov Jeffrey P. Bigham 42 22 0 10 Jun 2021
A Comprehensive Assessment of Dialog Evaluation Metrics Yi-Ting Yeh M. Eskénazi Shikib Mehri 36 104 0 07 Jun 2021
HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversations Weixin Liang Kai-Hui Liang Zhou Yu 39 15 0 01 Jun 2021
Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey Jinjie Ni Tom Young Vlad Pandelea Fuzhao Xue Min Zhang 54 268 0 10 May 2021
LEGOEval: An Open-Source Toolkit for Dialogue System Evaluation via Crowdsourcing Yu Li Josh Arnold Feifan Yan Weiyan Shi Zhou Yu ELM 31 11 0 05 May 2021
Meta-evaluation of Conversational Search Evaluation Metrics Zeyang Liu K. Zhou Max L. Wilson ELM 32 17 0 27 Apr 2021
Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems E. Razumovskaia Goran Glavavs Olga Majewska Edoardo Ponti Anna Korhonen Ivan Vulić 26 32 0 17 Apr 2021
$$Q^{2}$: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering$ $Q^{2}$ : Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering Or Honovich Leshem Choshen Roee Aharoni Ella Neeman Idan Szpektor Omri Abend HILM 36 138 0 16 Apr 2021
OpenViDial: A Large-Scale, Open-Domain Dialogue Dataset with Visual Contexts Yuxian Meng Shuhe Wang Qinghong Han Xiaofei Sun Fei Wu Rui Yan Jiwei Li 27 28 0 30 Dec 2020
Deconstruct to Reconstruct a Configurable Evaluation Metric for Open-Domain Dialogue Systems Vitou Phy Yang Zhao Akiko Aizawa 14 55 0 01 Nov 2020
An Evaluation Protocol for Generative Conversational Systems Seolhwa Lee Heuiseok Lim Jo˜ao Sedoc ELM 35 10 0 24 Oct 2020
Predicting User Engagement Status for Online Evaluation of Intelligent Assistants Rui Meng Zhen Yue A. Glass 21 2 0 01 Oct 2020
UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation Jian Guan Minlie Huang 29 69 0 16 Sep 2020
A Survey of Evaluation Metrics Used for NLG Systems Ananya B. Sai Akash Kumar Mohankumar Mitesh M. Khapra ELM 33 230 0 27 Aug 2020
Evaluation of Text Generation: A Survey Asli Celikyilmaz Elizabeth Clark Jianfeng Gao ELM LM&MA 19 377 0 26 Jun 2020
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions Stephen Roller Y-Lan Boureau Jason Weston Antoine Bordes Emily Dinan ... Kurt Shuster Eric Michael Smith Arthur Szlam Jack Urbanek Mary Williamson LLMAG AI4CE 28 51 0 22 Jun 2020
Report from the NSF Future Directions Workshop, Toward User-Oriented Agents: Research Directions and Challenges M. Eskénazi Tiancheng Zhao LLMAG AI4TS AI4CE 36 9 0 10 Jun 2020
Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluation Weixin Liang James Zou Zhou Yu ELM 34 33 0 21 May 2020
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization Esin Durmus He He Mona T. Diab HILM 23 384 0 07 May 2020
Learning an Unreferenced Metric for Online Dialogue Evaluation Koustuv Sinha Prasanna Parthasarathi Jasmine Wang Ryan J. Lowe William L. Hamilton Joelle Pineau OffRL 29 84 0 01 May 2020
Stay Hungry, Stay Focused: Generating Informative and Specific Questions in Information-Seeking Conversations Peng Qi Yuhao Zhang Christopher D. Manning 21 38 0 30 Apr 2020
A Survey of Document Grounded Dialogue Systems (DGDS) Longxuan Ma Weinan Zhang Mingda Li Ting Liu 32 19 0 17 Apr 2020