Human and LLM Biases in Hate Speech Annotations: A Socio-Demographic Analysis of Annotators and Targets

10 October 2024

Papers citing "Human and LLM Biases in Hate Speech Annotations: A Socio-Demographic Analysis of Annotators and Targets"

27 / 27 papers shown

Title
Bias in Decision-Making for AI's Ethical Dilemmas: A Comparative Study of ChatGPT and Claude Yile Yan Yinlin Zhu Wentao Xu ELM 75 0 0 17 Jan 2025
Correcting Annotator Bias in Training Data: Population-Aligned Instance Replication (PAIR) Stephanie Eckman Bolei Ma Christoph Kern Rob Chew Yun Xue Frauke Kreuter 53 0 0 12 Jan 2025
Generative Agent Simulations of 1,000 People Joon Sung Park Carolyn Q. Zou Aaron Shaw Benjamin Mako Hill Carrie J. Cai Meredith Ringel Morris Robb Willer Percy Liang Michael S. Bernstein SyDa VGen LM&Ro AI4CE 37 95 0 15 Nov 2024
Which Demographics do LLMs Default to During Annotation? Johannes Schäfer Aidan Combs Christopher Bagdon Jiahui Li Nadine Probol ... Yarik Menchaca Resendiz Aswathy Velutharambath Amelie Wuhrl Sabine Weber Roman Klinger 60 2 0 11 Oct 2024
Y Social: an LLM-powered Social Media Digital Twin Yaroslav Marchukov Massimo Stella Rémy Cazabet Katherine Abramski Erica Cau Salvatore Citraro Andrea Failla Riccardo Improta Virginia Morini Valentina Pansanella SyDa 95 10 0 01 Aug 2024
Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization Yu-Min Tseng Yu-Chao Huang Teng-Yun Hsiao Yu-Ching Hsu Chao-Wei Huang Jia-Yin Foo Yun-Nung Chen LLMAG 328 86 0 03 Jun 2024
Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs Shashank Gupta Vaishnavi Shrivastava Ameet Deshpande Ashwin Kalyan Peter Clark Ashish Sabharwal Tushar Khot 163 114 0 08 Nov 2023
Generative Language Models Exhibit Social Identity Biases Tiancheng Hu Yara Kyrychenko Steve Rathje Nigel Collier S. V. D. Linden Jon Roozenbeek 63 43 0 24 Oct 2023
Simulating Social Media Using Large Language Models to Evaluate Alternative News Feed Algorithms Petter Törnberg D. Valeeva J. Uitermark Christopher Bail LLMAG 53 43 0 05 Oct 2023
Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Prompting Tilman Beck Hendrik Schuff Anne Lauscher Iryna Gurevych 69 36 0 13 Sep 2023
How Crowd Worker Factors Influence Subjective Annotations: A Study of Tagging Misogynistic Hate Speech in Tweets Danula Hettiachchi I. Holcombe-James Stephanie Livingstone Anjalee de Silva Matthew Lease Flora D. Salim Mark Sanderson 60 11 0 03 Sep 2023
Can Large Language Models Capture Dissenting Human Voices? Noah Lee Na Min An James Thorne ALM 85 32 0 23 May 2023
BiasAsker: Measuring the Bias in Conversational AI System Yuxuan Wan Wenxuan Wang Pinjia He Jiazhen Gu Haonan Bai Michael Lyu 41 68 0 21 May 2023
Can ChatGPT Assess Human Personalities? A General Evaluation Framework Haocong Rao Cyril Leung Chunyan Miao LLMAG LM&MA 47 80 0 01 Mar 2023
Discovering Language Model Behaviors with Model-Written Evaluations Ethan Perez Sam Ringer Kamilė Lukošiūtė Karina Nguyen Edwin Chen ... Danny Hernandez Deep Ganguli Evan Hubinger Nicholas Schiefer Jared Kaplan ALM 46 396 0 19 Dec 2022
Moral Mimicry: Large Language Models Produce Moral Rationalizations Tailored to Political Identity Gabriel Simmons 131 61 0 24 Sep 2022
Out of One, Many: Using Language Models to Simulate Human Samples Lisa P. Argyle Ethan C. Busby Nancy Fulda Joshua R Gubler Christopher Rytting David Wingate SyDa 75 588 0 14 Sep 2022
Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies Gati Aher RosaI. Arriaga Adam Tauman Kalai 100 389 0 18 Aug 2022
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models Aarohi Srivastava Abhinav Rastogi Abhishek Rao Abu Awal Md Shoeb Abubakar Abid ... Zhuoye Zhao Zijian Wang Zijie J. Wang Zirui Wang Ziyi Wu ELM 142 1,746 0 09 Jun 2022
Don't Blame the Annotator: Bias Already Starts in the Annotation Instructions Mihir Parmar Swaroop Mishra Mor Geva Chitta Baral 59 55 0 01 May 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 756 12,835 0 04 Mar 2022
Handling Bias in Toxic Speech Detection: A Survey Tanmay Garg Sarah Masud Tharun Suresh Tanmoy Chakraborty 56 96 0 26 Jan 2022
The Origin and Value of Disagreement Among Data Labelers: A Case Study of the Individual Difference in Hate Speech Annotation Yisi Sang Jeffrey Stanton 26 28 0 07 Dec 2021
Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection Maarten Sap Swabha Swayamdipta Laura Vianna Xuhui Zhou Yejin Choi Noah A. Smith 72 282 0 15 Nov 2021
On Releasing Annotator-Level Labels and Information in Datasets Vinodkumar Prabhakaran Aida Mostafazadeh Davani Mark Díaz 78 148 0 12 Oct 2021
Intersectional Bias in Hate Speech and Abusive Language Datasets Jae-Yeon Kim Carlos Ortiz S. Nam Sarah Santiago V. Datta 62 45 0 12 May 2020
Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets Mor Geva Yoav Goldberg Jonathan Berant 318 325 0 21 Aug 2019