The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographics

20 June 2023

Paul Röttger

Papers citing "The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographics"

20 / 20 papers shown

Title
Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptions Matthias Orlikowski Jiaxin Pei Paul Röttger Philipp Cimiano David Jurgens Dirk Hovy 59 1 0 28 Feb 2025
SubData: Bridging Heterogeneous Datasets to Enable Theory-Driven Evaluation of Political and Demographic Perspectives in LLMs Leon Fröhling Pietro Bernardelle Gianluca Demartini ALM 79 0 0 21 Dec 2024
Accurate and Data-Efficient Toxicity Prediction when Annotators Disagree Harbani Jaggi Kashyap Murali Eve Fleisig Erdem Bıyık 34 1 0 16 Oct 2024
Personas with Attitudes: Controlling LLMs for Diverse Data Annotation Leon Fröhling Gianluca Demartini Dennis Assenmacher 29 5 0 15 Oct 2024
Re-examining Sexism and Misogyny Classification with Annotator Attitudes Aiqi Jiang Nikolas Vitsakis Tanvi Dinkar Gavin Abercrombie Ioannis Konstas 47 1 0 04 Oct 2024
Hate Personified: Investigating the role of LLMs in content moderation Sarah Masud Sahajpreet Singh Viktor Hangya Alexander Fraser Tanmoy Chakraborty 30 7 0 03 Oct 2024
Let Guidelines Guide You: A Prescriptive Guideline-Centered Data Annotation Methodology Federico Ruggeri Eleonora Misino Arianna Muti Katerina Korre Paolo Torroni Alberto Barrón-Cedeño 39 0 0 20 Jun 2024
Annotation alignment: Comparing LLM and human annotations of conversational safety Rajiv Movva Pang Wei Koh Emma Pierson ALM 39 3 0 10 Jun 2024
Designing NLP Systems That Adapt to Diverse Worldviews Claudiu Creanga Liviu P. Dinu 18 0 0 18 May 2024
The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels Eve Fleisig Su Lin Blodgett Dan Klein Zeerak Talat 35 13 0 09 May 2024
Annotator-Centric Active Learning for Subjective NLP Tasks Michiel van der Meer Neele Falk P. Murukannaiah Enrico Liscio 36 8 0 24 Apr 2024
D3CODE: Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluation Aida Mostafazadeh Davani Mark Díaz Dylan K. Baker Vinodkumar Prabhakaran 34 8 0 16 Apr 2024
Emotion Analysis in NLP: Trends, Gaps and Roadmap for Future Directions Flor Miriam Plaza del Arco Alba Curry Amanda Cercas Curry Dirk Hovy 52 14 0 02 Mar 2024
Quantifying the Persona Effect in LLM Simulations Tiancheng Hu Nigel Collier 33 52 0 16 Feb 2024
Comparing Pre-trained Human Language Models: Is it Better with Human Context as Groups, Individual Traits, or Both? Nikita Soni Niranjan Balasubramanian H. A. Schwartz Dirk Hovy 36 3 0 23 Jan 2024
Capturing Perspectives of Crowdsourced Annotators in Subjective Learning Tasks Negar Mokhberian Myrl G. Marmarelis F. R. Hopp Valerio Basile Fred Morstatter Kristina Lerman 38 10 0 16 Nov 2023
GRASP: A Disagreement Analysis Framework to Assess Group Associations in Perspectives Vinodkumar Prabhakaran Christopher Homan Lora Aroyo Aida Mostafazadeh Davani Alicia Parrish Alex S. Taylor Mark Díaz Ding Wang Greg Serapio-García 45 9 0 09 Nov 2023
Architectural Sweet Spots for Modeling Human Label Variation by the Example of Argument Quality: It's Best to Relate Perspectives! Philipp Heinisch Matthias Orlikowski Julia Romberg Philipp Cimiano 15 3 0 06 Nov 2023
Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Prompting Tilman Beck Hendrik Schuff Anne Lauscher Iryna Gurevych 43 33 0 13 Sep 2023
When Do Annotator Demographics Matter? Measuring the Influence of Annotator Demographics with the POPQUORN Dataset Jiaxin Pei David Jurgens 40 31 0 12 Jun 2023