On Releasing Annotator-Level Labels and Information in Datasets

12 October 2021

Papers citing "On Releasing Annotator-Level Labels and Information in Datasets"

30 / 30 papers shown

Title
AI-Driven Automation Can Become the Foundation of Next-Era Science of Science Research Renqi Chen Haoyang Su Shixiang Tang Zhenfei Yin Qingbo Wu Hui Li Ye Sun Nanqing Dong Wanli Ouyang Philip Torr AI4CE 22 0 0 17 May 2025
AI Alignment at Your Discretion Maarten Buyl Hadi Khalaf C. M. Verdun Lucas Monteiro Paes Caio Vieira Machado Flavio du Pin Calmon 48 0 0 10 Feb 2025
Correcting Annotator Bias in Training Data: Population-Aligned Instance Replication (PAIR) Stephanie Eckman Bolei Ma Christoph Kern Rob Chew Yun Xue Frauke Kreuter 41 0 0 12 Jan 2025
Aggregation Artifacts in Subjective Tasks Collapse Large Language Models' Posteriors Georgios Chochlakis Alexandros Potamianos Kristina Lerman Shrikanth Narayanan 34 0 0 17 Oct 2024
Human and LLM Biases in Hate Speech Annotations: A Socio-Demographic Analysis of Annotators and Targets Tommaso Giorgi Lorenzo Cima T. Fagni Marco Avvenuti S. Cresci 45 10 0 10 Oct 2024
Bayesian WeakS-to-Strong from Text Classification to Generation Ziyun Cui Ziyang Zhang Wen Wu Wen Wu Chao Zhang 39 2 0 24 May 2024
Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback Vincent Conitzer Rachel Freedman J. Heitzig Wesley H. Holliday Bob M. Jacobs ... Eric Pacuit Stuart Russell Hailey Schoelkopf Emanuel Tewolde W. Zwicker 48 30 0 16 Apr 2024
Cost-Efficient Subjective Task Annotation and Modeling through Few-Shot Annotator Adaptation Preni Golazizian Ali Omrani Alireza S. Ziabari Morteza Dehghani 34 1 0 21 Feb 2024
GRASP: A Disagreement Analysis Framework to Assess Group Associations in Perspectives Vinodkumar Prabhakaran Christopher Homan Lora Aroyo Aida Mostafazadeh Davani Alicia Parrish Alex S. Taylor Mark Díaz Ding Wang Greg Serapio-García 45 9 0 09 Nov 2023
Semantic and Expressive Variation in Image Captions Across Languages Andre Ye Sebastin Santy Jena D. Hwang Amy X. Zhang Ranjay Krishna VLM 64 3 0 22 Oct 2023
Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Prompting Tilman Beck Hendrik Schuff Anne Lauscher Iryna Gurevych 43 34 0 13 Sep 2023
The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographics Matthias Orlikowski Paul Röttger Philipp Cimiano Italy 34 26 0 20 Jun 2023
Intersectionality in Conversational AI Safety: How Bayesian Multilevel Models Help Understand Diverse Perceptions of Safety Christopher Homan Greg Serapio-García Lora Aroyo Mark Díaz Alicia Parrish Vinodkumar Prabhakaran Alex S. Taylor Ding Wang 27 9 0 20 Jun 2023
What's the Meaning of Superhuman Performance in Today's NLU? Simone Tedeschi Johan Bos T. Declerck Jan Hajic Daniel Hershcovich ... Simon Krek Steven Schockaert Rico Sennrich Ekaterina Shutova Roberto Navigli ELM LM&MA VLM ReLM LRM 47 26 0 15 May 2023
When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks Eve Fleisig Rediet Abebe Dan Klein 34 44 0 11 May 2023
Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design Valentina Pyatkin Frances Yung Merel C. J. Scholman Reut Tsarfaty Ido Dagan Vera Demberg 27 12 0 03 Apr 2023
SemEval-2023 Task 10: Explainable Detection of Online Sexism Hannah Rose Kirk Wenjie Yin Bertie Vidgen Paul Röttger 24 117 0 07 Mar 2023
Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Information Ruyuan Wan Jaehyung Kim Dongyeop Kang 17 36 0 12 Jan 2023
The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation Barbara Plank 30 97 0 04 Nov 2022
Stop Measuring Calibration When Humans Disagree Joris Baan Wilker Aziz Barbara Plank Raquel Fernández 29 53 0 28 Oct 2022
Unifying Data Perspectivism and Personalization: An Application to Social Norms Joan Plepi Béla Neuendorf Lucie Flek Charles F Welch 46 21 0 26 Oct 2022
A Keyword Based Approach to Understanding the Overpenalization of Marginalized Groups by English Marginal Abuse Models on Twitter Kyra Yee Alice Schoenauer Sebag Olivia Redfield Emily Sheng Matthias Eck Luca Belli 28 2 0 07 Oct 2022
Re-contextualizing Fairness in NLP: The Case of India Shaily Bhatt Sunipa Dev Partha P. Talukdar Shachi Dave Vinodkumar Prabhakaran 32 54 0 25 Sep 2022
Don't Take it Personally: Analyzing Gender and Age Differences in Ratings of Online Humor J. A Meaney Steven R. Wilson Luis Chiruzzo Walid Magdy 16 3 0 23 Aug 2022
Eliciting and Learning with Soft Labels from Every Annotator K. M. Collins Umang Bhatt Adrian Weller 29 44 0 02 Jul 2022
From Stance to Concern: Adaptation of Propositional Analysis to New Tasks and Domains Brodie Mather Bonnie J. Dorr Adam Dalton William de Beaumont Owen Rambow Sonja M. Schmer-Galunder 30 8 0 20 Mar 2022
Challenges and Strategies in Cross-Cultural NLP Daniel Hershcovich Stella Frank Heather Lent Miryam de Lhoneux Mostafa Abdou ... Ruixiang Cui Constanza Fierro Katerina Margatina Phillip Rust Anders Søgaard 45 163 0 18 Mar 2022
Toward More Meaningful Resources for Lower-resourced Languages Constantine Lignos Nolan Holley Chester Palen-Michel Jonne Saleva 29 18 0 24 Feb 2022
Jury Learning: Integrating Dissenting Voices into Machine Learning Models Mitchell L. Gordon Michelle S. Lam J. Park Kayur Patel Jeffrey T. Hancock Tatsunori Hashimoto Michael S. Bernstein 27 146 0 07 Feb 2022
Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection Maarten Sap Swabha Swayamdipta Laura Vianna Xuhui Zhou Yejin Choi Noah A. Smith 46 268 0 15 Nov 2021