Judgment Sieve: Reducing Uncertainty in Group Judgments through Interventions Targeting Ambiguity versus Disagreement

2 May 2023

Papers citing "Judgment Sieve: Reducing Uncertainty in Group Judgments through Interventions Targeting Ambiguity versus Disagreement"

22 / 22 papers shown

Title
Eliciting and Learning with Soft Labels from Every Annotator Katherine M. Collins Umang Bhatt Adrian Weller 75 46 0 02 Jul 2022
In Search of Ambiguity: A Three-Stage Workflow Design to Clarify Annotation Guidelines for Crowd Workers V. Pradhan M. Schaekermann Matthew Lease 76 12 0 04 Dec 2021
Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection Maarten Sap Swabha Swayamdipta Laura Vianna Xuhui Zhou Yejin Choi Noah A. Smith 89 283 0 15 Nov 2021
On Releasing Annotator-Level Labels and Information in Datasets Vinodkumar Prabhakaran Aida Mostafazadeh Davani Mark Díaz 93 150 0 12 Oct 2021
Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators' Disagreement Elisa Leonardelli Stefano Menini Alessio Palmero Aprosio Marco Guerini Sara Tonelli 93 97 0 28 Sep 2021
Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment Corinna Cortes Neil D. Lawrence 101 46 0 20 Sep 2021
Goldilocks: Consistent Crowdsourced Scalar Annotations with Relative Uncertainty Quan Ze Chen Daniel S. Weld Amy X. Zhang 61 17 0 04 Aug 2021
Annotation Curricula to Implicitly Train Non-Expert Annotators Ji-Ung Lee Jan-Christoph Klie Iryna Gurevych 44 11 0 04 Jun 2021
The Challenge of Variable Effort Crowdsourcing and How Visible Gold Can Help Danula Hettiachchi M. Schaekermann Tristan McKinney Matthew Lease 81 20 0 20 May 2021
Re-TACRED: Addressing Shortcomings of the TACRED Dataset George Stoica Emmanouil Antonios Platanios Barnabas Poczos 80 116 0 16 Apr 2021
Abuse is Contextual, What about NLP? The Role of Context in Abusive Language Annotation and Detection Stefano Menini Alessio Palmero Aprosio Sara Tonelli 63 40 0 27 Mar 2021
Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty Umang Bhatt Javier Antorán Yunfeng Zhang Q. V. Liao P. Sattigeri ... L. Nachman R. Chunara Madhulika Srikumar Adrian Weller Alice Xiang 97 251 0 15 Nov 2020
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics Swabha Swayamdipta Roy Schwartz Nicholas Lourie Yizhong Wang Hannaneh Hajishirzi Noah A. Smith Yejin Choi 140 452 0 22 Sep 2020
Are we done with ImageNet? Lucas Beyer Olivier J. Hénaff Alexander Kolesnikov Xiaohua Zhai Aaron van den Oord VLM 134 407 0 12 Jun 2020
Toxicity Detection: Does Context Really Matter? John Pavlopoulos Jeffrey Scott Sorensen Lucas Dixon Nithum Thain Ion Androutsopoulos 87 168 0 01 Jun 2020
Garbage In, Garbage Out? Do Machine Learning Application Papers in Social Computing Report Where Human-Labeled Training Data Comes From? R. Geiger Kevin Yu Yanlai Yang Mindy Dai Jie Qiu Rebekah Tang Jenny Huang 64 153 0 17 Dec 2019
Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods Eyke Hüllermeier Willem Waegeman PER UD 258 1,432 0 21 Oct 2019
Datasheets for Datasets Timnit Gebru Jamie Morgenstern Briana Vecchione Jennifer Wortman Vaughan Hanna M. Wallach Hal Daumé Kate Crawford 294 2,201 0 23 Mar 2018
Crowdsourcing Ground Truth for Medical Relation Extraction Anca Dumitrache Lora Aroyo Chris Welty 67 65 0 09 Jan 2017
ImageNet Large Scale Visual Recognition Challenge Olga Russakovsky Jia Deng Hao Su J. Krause S. Satheesh ... A. Karpathy A. Khosla Michael S. Bernstein Alexander C. Berg Li Fei-Fei VLM ObjD 1.7K 39,637 0 01 Sep 2014
Sockpuppet Detection in Wikipedia: A Corpus of Real-World Deceptive Writing for Linking Identities Thamar Solorio Ragib Hasan M. Mizan 81 40 0 24 Oct 2013
Online Deliberation Design: Choices, Criteria, and Evidence Todd R. Davies Reid Chandler 63 69 0 21 Feb 2013