Rater Cohesion and Quality from a Vicarious Perspective

15 August 2024

Deepak Pandita

Tharindu Cyril Weerasooriya

Sujan Dutta

Sarah K. K. Luger

Tharindu Ranasinghe

Ashiqur R. KhudaBukhsh

Marcos Zampieri

Christopher M. Homan

ArXiv (abs)PDF HTML

Papers citing "Rater Cohesion and Quality from a Vicarious Perspective"

21 / 21 papers shown

Title
GRASP: A Disagreement Analysis Framework to Assess Group Associations in Perspectives Vinodkumar Prabhakaran Christopher Homan Lora Aroyo Aida Mostafazadeh Davani Alicia Parrish Alex S. Taylor Mark Díaz Ding Wang Greg Serapio-García 82 9 0 09 Nov 2023
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback Stephen Casper Xander Davies Claudia Shi T. Gilbert Jérémy Scheurer ... Erdem Biyik Anca Dragan David M. Krueger Dorsa Sadigh Dylan Hadfield-Menell ALM OffRL 135 530 0 27 Jul 2023
Intersectionality in Conversational AI Safety: How Bayesian Multilevel Models Help Understand Diverse Perceptions of Safety Christopher Homan Greg Serapio-García Lora Aroyo Mark Díaz Alicia Parrish Vinodkumar Prabhakaran Alex S. Taylor Ding Wang 71 9 0 20 Jun 2023
When Do Annotator Demographics Matter? Measuring the Influence of Annotator Demographics with the POPQUORN Dataset Jiaxin Pei David Jurgens 68 34 0 12 Jun 2023
From Dogwhistles to Bullhorns: Unveiling Coded Rhetoric with Language Models Julia Mendelsohn Ronan Le Bras Yejin Choi Maarten Sap 43 29 0 26 May 2023
Vicarious Offense and Noise Audit of Offensive Speech Classifiers: Unifying Human and Machine Disagreement on What is Offensive Tharindu Cyril Weerasooriya Sujan Dutta Tharindu Ranasinghe Marcos Zampieri Christopher Homan Ashiqur R. KhudaBukhsh AAML 81 21 0 29 Jan 2023
Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotation Nitesh Goyal Ian D Kivlichan Rachel Rosen Lucy Vasserman 80 93 0 01 May 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 886 13,207 0 04 Mar 2022
Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection Maarten Sap Swabha Swayamdipta Laura Vianna Xuhui Zhou Yejin Choi Noah A. Smith 81 283 0 15 Nov 2021
On Releasing Annotator-Level Labels and Information in Datasets Vinodkumar Prabhakaran Aida Mostafazadeh Davani Mark Díaz 90 150 0 12 Oct 2021
Toward a Perspectivist Turn in Ground Truthing for Predictive Computing Valerio Basile F. Cabitza Andrea Campagner Michael Fell 72 161 0 09 Sep 2021
Cross-replication Reliability -- An Empirical Approach to Interpreting Inter-rater Reliability KayYen Wong Praveen K. Paritosh Lora Aroyo 45 28 0 11 Jun 2021
Designing Toxic Content Classification for a Diversity of Perspectives Deepak Kumar Patrick Gage Kelley Sunny Consolvo Joshua Mason Elie Bursztein Zakir Durumeric Kurt Thomas Michael C. Bailey 51 109 0 04 Jun 2021
We Don't Speak the Same Language: Interpreting Polarization through Machine Translation Ashiqur R. KhudaBukhsh Rupak Sarkar M. Kamlet Tom Michael Mitchell 34 51 0 05 Oct 2020
Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets Mor Geva Yoav Goldberg Jonathan Berant 323 326 0 21 Aug 2019
Racial Bias in Hate Speech and Abusive Language Detection Datasets Thomas Davidson Debasmita Bhattacharya Ingmar Weber 109 459 0 29 May 2019
Reducing Gender Bias in Abusive Language Detection Ji Ho Park Jamin Shin Pascale Fung FaML 51 341 0 22 Aug 2018
CrowdTruth 2.0: Quality Metrics for Crowdsourcing with Disagreement Anca Dumitrache Oana Inel Lora Aroyo Benjamin Timmermans Chris Welty HILM 43 47 0 18 Aug 2018
Argument Strength is in the Eye of the Beholder: Audience Effects in Persuasion S. Lukin P. Anand M. Walker S. Whittaker 57 109 0 30 Aug 2017
Like trainer, like bot? Inheritance of bias in algorithmic content moderation Reuben Binns Michael Veale Max Van Kleek N. Shadbolt 44 149 0 05 Jul 2017
Interactive Learning from Policy-Dependent Human Feedback J. MacGlashan Mark K. Ho R. Loftin Bei Peng Guan Wang David L. Roberts Matthew E. Taylor Michael L. Littman 87 306 0 21 Jan 2017