Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents

12 January 2022

Jason Weston

Papers citing "Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents"

40 / 40 papers shown

Title
Evaluation and Facilitation of Online Discussions in the LLM Era: A Survey Katerina Korre Dimitris Tsirmpas Nikos Gkoumas Emma Cabalé Danai Myrtzani Theodoros Evgeniou Ion Androutsopoulos Ion Androutsopoulos 81 2 0 03 Mar 2025
LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts Helia Hashemi J. Eisner Corby Rosset Benjamin Van Durme Chris Kedzie 109 4 0 03 Jan 2025
A Survey of NLP-Related Crowdsourcing HITs: what works and what does not Jessica Huynh Jeffrey P. Bigham M. Eskénazi 84 18 0 09 Nov 2021
Reason first, then respond: Modular Generation for Knowledge-infused Dialogue Leonard Adolphs Kurt Shuster Jack Urbanek Arthur Szlam Jason Weston KELM LRM 246 41 0 09 Nov 2021
Beyond Goldfish Memory: Long-Term Open-Domain Conversation Jing Xu Arthur Szlam Jason Weston RALM 58 251 0 15 Jul 2021
Internet-Augmented Dialogue Generation M. Komeili Kurt Shuster Jason Weston RALM 297 287 0 15 Jul 2021
A Comprehensive Assessment of Dialog Evaluation Metrics Yi-Ting Yeh M. Eskénazi Shikib Mehri 66 109 0 07 Jun 2021
An Evaluation Protocol for Generative Conversational Systems Seolhwa Lee Heuiseok Lim Jo˜ao Sedoc ELM 75 10 0 24 Oct 2020
Recipes for Safety in Open-domain Chatbots Jing Xu Da Ju Margaret Li Y-Lan Boureau Jason Weston Emily Dinan 72 234 0 14 Oct 2020
Spot The Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems Jan Deriu Don Tuggener Pius von Daniken Jon Ander Campos Álvaro Rodrigo Thiziri Belkacem Aitor Soroa Etxabe Eneko Agirre Mark Cieliebak 41 36 0 05 Oct 2020
Deploying Lifelong Open-Domain Dialogue Learning Kurt Shuster Jack Urbanek Emily Dinan Arthur Szlam Jason Weston 66 22 0 18 Aug 2020
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions Stephen Roller Y-Lan Boureau Jason Weston Antoine Bordes Emily Dinan ... Kurt Shuster Eric Michael Smith Arthur Szlam Jack Urbanek Mary Williamson LLMAG AI4CE 92 52 0 22 Jun 2020
Towards Unified Dialogue System Evaluation: A Comprehensive Analysis of Current Evaluation Protocols Sarah E. Finch Jinho Choi ELM 61 67 0 10 Jun 2020
Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluation Weixin Liang James Zou Zhou Yu ELM 75 34 0 21 May 2020
Recipes for building an open-domain chatbot Stephen Roller Emily Dinan Naman Goyal Da Ju Mary Williamson ... Myle Ott Kurt Shuster Eric Michael Smith Y-Lan Boureau Jason Weston ALM 123 1,009 0 28 Apr 2020
Towards a Human-like Open-Domain Chatbot Daniel De Freitas Minh-Thang Luong David R. So Jamie Hall Noah Fiedel ... Zi Yang Apoorv Kulshreshtha Gaurav Nemade Yifeng Lu Quoc V. Le 114 938 0 27 Jan 2020
ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons Margaret Li Jason Weston Stephen Roller 64 177 0 06 Sep 2019
Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack Emily Dinan Samuel Humeau Bharath Chintagunta Jason Weston 84 247 0 17 Aug 2019
Neural Text Generation with Unlikelihood Training Sean Welleck Ilia Kulikov Stephen Roller Emily Dinan Kyunghyun Cho Jason Weston MU 50 578 0 12 Aug 2019
Investigating Evaluation of Open-Domain Dialogue Systems With Human Generated Multiple References Prakhar Gupta Shikib Mehri Tiancheng Zhao Amy Pavel M. Eskénazi Jeffrey P. Bigham 58 87 0 24 Jul 2019
ELI5: Long Form Question Answering Angela Fan Yacine Jernite Ethan Perez David Grangier Jason Weston Michael Auli AI4MH ELM 92 619 0 22 Jul 2019
Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems Asma Ghandeharioun J. Shen Natasha Jaques Craig Ferguson Noah J. Jones Àgata Lapedriza Rosalind W. Picard 57 91 0 21 Jun 2019
Challenges in Building Intelligent Open-domain Dialog Systems Minlie Huang Xiaoyan Zhu Jianfeng Gao VLM 100 316 0 13 May 2019
Survey on Evaluation Methods for Dialogue Systems Jan Deriu Álvaro Rodrigo Arantxa Otegi Guillermo Echegoyen S. Rosset Eneko Agirre Mark Cieliebak 73 284 0 10 May 2019
Unifying Human and Statistical Evaluation for Natural Language Generation Tatsunori B. Hashimoto Hugh Zhang Percy Liang 72 224 0 04 Apr 2019
What makes a good conversation? How controllable attributes affect human judgments A. See Stephen Roller Douwe Kiela Jason Weston 99 288 0 22 Feb 2019
The Second Conversational Intelligence Challenge (ConvAI2) Emily Dinan V. Logacheva Valentin Malykh Alexander H. Miller Kurt Shuster ... Alexander I. Rudnicky Jason Williams Joelle Pineau Andrey Kravchenko Jason Weston DRL 99 367 0 31 Jan 2019
Wizard of Wikipedia: Knowledge-Powered Conversational agents Emily Dinan Stephen Roller Kurt Shuster Angela Fan Michael Auli Jason Weston RALM KELM 129 950 0 03 Nov 2018
Importance of Search and Evaluation Strategies in Neural Dialogue Modeling Ilia Kulikov Alexander H. Miller Kyunghyun Cho Jason Weston 72 84 0 02 Nov 2018
Response Generation by Context-aware Prototype Editing Yu Wu Furu Wei Shaohan Huang Yunli Wang Zhoujun Li Ming Zhou KELM 61 121 0 19 Jun 2018
Personalizing Dialogue Agents: I have a dog, do you have pets too? Saizheng Zhang Emily Dinan Jack Urbanek Arthur Szlam Douwe Kiela Jason Weston 105 1,459 0 22 Jan 2018
On Evaluating and Comparing Open Domain Dialog Systems Anu Venkatesh Chandra Khatri A. Ram Fenfei Guo Raefer Gabriel ... Behnam Hedayatnia A. Metallinou Rahul Goel Shaohua Yang A. Raju LLMAG ELM 65 82 0 11 Jan 2018
Conversational AI: The Science Behind the Alexa Prize A. Ram R. Prasad Chandra Khatri Anu Venkatesh Raefer Gabriel ... Yi Pan Han Song Sk Jayadevan Gene Hwang Art Pettigrue 78 282 0 11 Jan 2018
Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses Ryan J. Lowe Michael Noseworthy Iulian Serban Nicolas Angelard-Gontier Yoshua Bengio Joelle Pineau 57 372 0 23 Aug 2017
Frames: A Corpus for Adding Memory to Goal-Oriented Dialogue Systems Layla El Asri Hannes Schulz Shikhar Sharma Jeremie Zumer Justin Harris Emery Fine Rahul Mehrotra Kaheer Suleman 127 270 0 31 Mar 2017
Learning End-to-End Goal-Oriented Dialog Antoine Bordes Y-Lan Boureau Jason Weston 82 782 0 24 May 2016
A Network-based End-to-End Trainable Task-oriented Dialogue System Tsung-Hsien Wen David Vandyke N. Mrksic Milica Gasic L. Rojas-Barahona Pei-hao Su Stefan Ultes S. Young 74 1,108 0 15 Apr 2016
How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation Chia-Wei Liu Ryan J. Lowe Iulian Serban Michael Noseworthy Laurent Charlin Joelle Pineau 104 1,297 0 25 Mar 2016
A Persona-Based Neural Conversation Model Jiwei Li Michel Galley Chris Brockett Georgios P. Spithourakis Jianfeng Gao W. Dolan 116 1,036 0 19 Mar 2016
A Neural Conversational Model Oriol Vinyals Quoc V. Le BDL 137 1,767 0 19 Jun 2015