Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Information

12 January 2023

Papers citing "Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Information"

24 / 24 papers shown

Title
Conflicts in Texts: Data, Implications and Challenges Siyi Liu Dan Roth 166 0 0 28 Apr 2025
CLASH: Evaluating Language Models on Judging High-Stakes Dilemmas from Multiple Perspectives Ayoung Lee Ryan Sungmo Kwon Peter Railton Lu Wang ELM 51 0 0 15 Apr 2025
Evaluating how LLM annotations represent diverse views on contentious topics Megan A. Brown Shubham Atreja Libby Hemphill Patrick Y. Wu 158 0 0 29 Mar 2025
QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks? Belinda Z. Li Been Kim Zehao Wang LRM 43 2 0 28 Mar 2025
Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptions Matthias Orlikowski Jiaxin Pei Paul Röttger Philipp Cimiano David Jurgens Dirk Hovy 59 1 0 28 Feb 2025
Leveraging Annotator Disagreement for Text Classification Jin Xu Mariët Theune Daniel Braun 32 2 0 26 Sep 2024
Does Cross-Cultural Alignment Change the Commonsense Morality of Language Models? Yuu Jinnai 49 1 0 24 Jun 2024
A Taxonomy of Challenges to Curating Fair Datasets Dora Zhao M. Scheuerman Pooja Chitre Jerone T. A. Andrews Georgia Panagiotidou Shawn Walker Kathleen H. Pine Alice Xiang 47 2 0 10 Jun 2024
Annotation alignment: Comparing LLM and human annotations of conversational safety Rajiv Movva Pang Wei Koh Emma Pierson ALM 39 3 0 10 Jun 2024
Annotation-Efficient Preference Optimization for Language Model Alignment Yuu Jinnai Ukyo Honda 42 0 0 22 May 2024
The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels Eve Fleisig Su Lin Blodgett Dan Klein Zeerak Talat 32 13 0 09 May 2024
Beyond Accuracy: Investigating Error Types in GPT-4 Responses to USMLE Questions Soumyadeep Roy A. Khatua Fatemeh Ghoochani Uwe Hadler Wolfgang Nejdl Niloy Ganguly ELM LM&MA 46 8 0 20 Apr 2024
Don't Blame the Data, Blame the Model: Understanding Noise and Bias When Learning from Subjective Annotations Abhishek Anand Negar Mokhberian Prathyusha Naresh Kumar Anweasha Saha Zihao He Ashwin Rao Fred Morstatter Kristina Lerman 39 6 0 06 Mar 2024
On Diversified Preferences of Large Language Model Alignment Dun Zeng Yong Dai Pengyu Cheng Longyue Wang Tianhao Hu Wanshun Chen Nan Du Zenglin Xu ALM 38 16 0 12 Dec 2023
Disentangling Perceptions of Offensiveness: Cultural and Moral Correlates Aida Mostafazadeh Davani Mark Díaz Dylan K. Baker Vinodkumar Prabhakaran AAML 31 14 0 11 Dec 2023
Capturing Perspectives of Crowdsourced Annotators in Subjective Learning Tasks Negar Mokhberian Myrl G. Marmarelis F. R. Hopp Valerio Basile Fred Morstatter Kristina Lerman 38 10 0 16 Nov 2023
Unveiling the Multi-Annotation Process: Examining the Influence of Annotation Quantity and Instance Difficulty on Model Performance Pritam Kadasi Mayank Singh 29 3 0 23 Oct 2023
Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Prompting Tilman Beck Hendrik Schuff Anne Lauscher Iryna Gurevych 43 33 0 13 Sep 2023
When Do Annotator Demographics Matter? Measuring the Influence of Annotator Demographics with the POPQUORN Dataset Jiaxin Pei David Jurgens 40 31 0 12 Jun 2023
NLPositionality: Characterizing Design Biases of Datasets and Models Sebastin Santy Jenny T Liang Ronan Le Bras Katharina Reinecke Maarten Sap 32 77 0 02 Jun 2023
Annotation Imputation to Individualize Predictions: Initial Studies on Distribution Dynamics and Model Predictions London Lowmanstone Ruyuan Wan Risako Owan Jaehyung Kim Dongyeop Kang 24 1 0 24 May 2023
Sociocultural Norm Similarities and Differences via Situational Alignment and Explainable Textual Entailment Sky CH-Wang Arkadiy Saakyan Aochong Li Zhou Yu Smaranda Muresan 29 16 0 23 May 2023
When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks Eve Fleisig Rediet Abebe Dan Klein 34 43 0 11 May 2023
DynaSent: A Dynamic Benchmark for Sentiment Analysis Christopher Potts Zhengxuan Wu Atticus Geiger Douwe Kiela 230 77 0 30 Dec 2020