Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection

15 November 2021

Maarten Sap

Swabha Swayamdipta

Laura Vianna

Xuhui Zhou

Yejin Choi

Noah A. Smith

ArXiv PDF HTML

Papers citing "Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection"

50 / 165 papers shown

Title
COBIAS: Assessing the Contextual Reliability of Bias Benchmarks for Language Models Priyanshul Govil Hemang Jain Vamshi Krishna Bonagiri Aman Chadha Ponnurangam Kumaraguru Manas Gaur Sanorita Dey 53 2 0 22 Feb 2024
Wikibench: Community-Driven Data Curation for AI Evaluation on Wikipedia Tzu-Sheng Kuo Aaron L Halfaker Zirui Cheng Jiwoo Kim Meng-Hsin Wu Tongshuang Wu Kenneth Holstein Haiyi Zhu 62 21 0 21 Feb 2024
Cost-Efficient Subjective Task Annotation and Modeling through Few-Shot Annotator Adaptation Preni Golazizian Ali Omrani Alireza S. Ziabari Morteza Dehghani 23 1 0 21 Feb 2024
Infrastructure Ombudsman: Mining Future Failure Concerns from Structural Disaster Response Towhid Chowdhury Soumyajit Datta Naveen Sharma Ashiqur R. KhudaBukhsh AI4CE 34 4 0 21 Feb 2024
Understanding Fine-grained Distortions in Reports of Scientific Findings Amelie Wuhrl Dustin Wright Roman Klinger Isabelle Augenstein 35 3 0 19 Feb 2024
Quantifying the Persona Effect in LLM Simulations Tiancheng Hu Nigel Collier 33 52 0 16 Feb 2024
MaxMin-RLHF: Towards Equitable Alignment of Large Language Models with Diverse Human Preferences Souradip Chakraborty Jiahao Qiu Hui Yuan Alec Koppel Furong Huang Dinesh Manocha Amrit Singh Bedi Mengdi Wang ALM 30 47 0 14 Feb 2024
Through the Lens of Split Vote: Exploring Disagreement, Difficulty and Calibration in Legal Case Outcome Classification Shanshan Xu Santosh T.Y.S.S O. Ichim Barbara Plank Matthias Grabmair 37 4 0 11 Feb 2024
Discipline and Label: A WEIRD Genealogy and Social Theory of Data Annotation Andrew Smart Ding Wang Ellis Monk Mark Díaz Atoosa Kasirzadeh Erin van Liemt Sonja Schmer-Galunder 36 8 0 09 Feb 2024
Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertainty Kaitlyn Zhou Jena D. Hwang Xiang Ren Maarten Sap 36 54 0 12 Jan 2024
Quantifying the Uniqueness of Donald Trump in Presidential Discourse Karen Zhou Alexander A. Meitus Milo Chase Grace Wang Anne Mykland William Howell Chenhao Tan 17 1 0 02 Jan 2024
Understanding News Creation Intents: Frame, Dataset, and Method Zhengjia Wang Danding Wang Qiang Sheng Juan Cao Silong Su Yifan Sun Beizhe Hu Siyuan Ma 21 4 0 27 Dec 2023
Disentangling Perceptions of Offensiveness: Cultural and Moral Correlates Aida Mostafazadeh Davani Mark Díaz Dylan K. Baker Vinodkumar Prabhakaran AAML 23 14 0 11 Dec 2023
A Block Metropolis-Hastings Sampler for Controllable Energy-based Text Generation Jarad Forristal Niloofar Mireshghallah Greg Durrett Taylor Berg-Kirkpatrick 115 4 0 07 Dec 2023
Automatic Construction of a Korean Toxic Instruction Dataset for Ethical Tuning of Large Language Models Sungjoo Byun Dongjun Jang Hyemi Jo Hyopil Shin 24 2 0 30 Nov 2023
A Survey of the Evolution of Language Model-Based Dialogue Systems Hongru Wang Lingzhi Wang Yiming Du Liang Chen Jing Zhou Yufei Wang Kam-Fai Wong LRM 59 20 0 28 Nov 2023
Capturing Perspectives of Crowdsourced Annotators in Subjective Learning Tasks Negar Mokhberian Myrl G. Marmarelis F. R. Hopp Valerio Basile Fred Morstatter Kristina Lerman 32 9 0 16 Nov 2023
P^3SUM: Preserving Author's Perspective in News Summarization with Diffusion Language Models Yuhan Liu Shangbin Feng Xiaochuang Han Vidhisha Balachandran Chan Young Park Sachin Kumar Yulia Tsvetkov DiffM 41 2 0 16 Nov 2023
A Survey of Confidence Estimation and Calibration in Large Language Models Jiahui Geng Fengyu Cai Yuxia Wang Heinz Koeppl Preslav Nakov Iryna Gurevych UQCV 41 54 0 14 Nov 2023
Gen-Z: Generative Zero-Shot Text Classification with Contextualized Label Descriptions Sachin Kumar Chan Young Park Yulia Tsvetkov VLM 24 2 0 13 Nov 2023
GRASP: A Disagreement Analysis Framework to Assess Group Associations in Perspectives Vinodkumar Prabhakaran Christopher Homan Lora Aroyo Aida Mostafazadeh Davani Alicia Parrish Alex S. Taylor Mark Díaz Ding Wang Greg Serapio-García 37 9 0 09 Nov 2023
Dimensions of Online Conflict: Towards Modeling Agonism Matt Canute Mali Jin hannah holtzclaw Alberto Lusoli Philippa R Adams Mugdha Pandya Maite Taboada Diana Maynard Wendy Hui Kyong Chun 13 1 0 06 Nov 2023
Beyond Denouncing Hate: Strategies for Countering Implied Biases and Stereotypes in Language Jimin Mun Emily Allaway Akhila Yerukola Laura Vianna Sarah-Jane Leslie Maarten Sap 16 22 0 31 Oct 2023
Defining a New NLP Playground Sha Li Chi Han Pengfei Yu Carl N. Edwards Manling Li ... Yi Ren Fung Charles Yu Joel R. Tetreault Eduard H. Hovy Heng Ji 35 5 0 31 Oct 2023
Representativeness as a Forgotten Lesson for Multilingual and Code-switched Data Collection and Preparation A. Seza Doğruöz Sunayana Sitaram Zheng-Xin Yong 27 13 0 31 Oct 2023
ACTOR: Active Learning with Annotator-specific Classification Heads to Embrace Human Label Variation Xinpeng Wang Barbara Plank 11 6 0 23 Oct 2023
Unveiling the Multi-Annotation Process: Examining the Influence of Annotation Quantity and Instance Difficulty on Model Performance Pritam Kadasi Mayank Singh 21 3 0 23 Oct 2023
From Dissonance to Insights: Dissecting Disagreements in Rationale Construction for Case Outcome Classification Shanshan Xu Santosh T.Y.S.S O. Ichim Isabella Risini Barbara Plank Matthias Grabmair AILaw 38 12 0 18 Oct 2023
VECHR: A Dataset for Explainable and Robust Classification of Vulnerability Type in the European Court of Human Rights Shanshan Xu Leon Staufer Santosh T.Y.S.S O. Ichim Corina Heri Matthias Grabmair 18 0 0 17 Oct 2023
Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations Zhuoyan Li Hangxiao Zhu Zhuoran Lu Ming Yin SyDa 69 67 0 11 Oct 2023
The Empty Signifier Problem: Towards Clearer Paradigms for Operationalising "Alignment" in Large Language Models Hannah Rose Kirk Bertie Vidgen Paul Röttger Scott A. Hale 44 2 0 03 Oct 2023
On the definition of toxicity in NLP Sergey Berezin R. Farahbakhsh Noel Crespi 21 0 0 03 Oct 2023
PopBERT. Detecting populism and its host ideologies in the German Bundestag Lukas Erhard Sara Hanke Uwe Remer A. Falenska R. Heiberger 25 2 0 22 Sep 2023
Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Prompting Tilman Beck Hendrik Schuff Anne Lauscher Iryna Gurevych 37 32 0 13 Sep 2023
On the Challenges of Building Datasets for Hate Speech Detection Vitthal Bhandari 18 1 0 06 Sep 2023
Donkii: Can Annotation Error Detection Methods Find Errors in Instruction-Tuning Datasets? Leon Weber-Genzel Robert Litschko Ekaterina Artemova Barbara Plank 18 2 0 04 Sep 2023
How Crowd Worker Factors Influence Subjective Annotations: A Study of Tagging Misogynistic Hate Speech in Tweets Danula Hettiachchi I. Holcombe-James Stephanie Livingstone Anjalee de Silva Matthew Lease Flora D. Salim Mark Sanderson 18 10 0 03 Sep 2023
Exploring Cross-Cultural Differences in English Hate Speech Annotations: From Dataset Construction to Analysis Nayeon Lee Chani Jung Jun-Hee Myung Jiho Jin Jose Camacho-Collados Juho Kim Alice H. Oh 44 14 0 31 Aug 2023
Rethinking Machine Ethics -- Can LLMs Perform Moral Reasoning through the Lens of Moral Theories? Jingyan Zhou Minda Hu Junan Li Xiaoying Zhang Xixin Wu Irwin King Helen M. Meng LRM 42 24 0 29 Aug 2023
BAN-PL: a Novel Polish Dataset of Banned Harmful and Offensive Content from Wykop.pl web service Anna Kołos Inez Okulska Kinga Głąbińska Agnieszka Karlinska Emilia Wisnios Paweł Ellerik Andrzej Prałat 11 1 0 21 Aug 2023
Mitigating Voter Attribute Bias for Fair Opinion Aggregation Ryosuke Ueda Koh Takeuchi H. Kashima 8 1 0 20 Jul 2023
Subjective Crowd Disagreements for Subjective Data: Uncovering Meaningful CrowdOpinion with Population-level Learning Tharindu Cyril Weerasooriya Sarah K. K. Luger Saloni Poddar Ashiqur R. KhudaBukhsh Christopher Homan 15 5 0 07 Jul 2023
Understanding Counterspeech for Online Harm Mitigation Yi-Ling Chung Gavin Abercrombie Florence E. Enock Jonathan Bright Verena Rieser 25 16 0 01 Jul 2023
Towards Measuring the Representation of Subjective Global Opinions in Language Models Esin Durmus Karina Nyugen Thomas I. Liao Nicholas Schiefer Amanda Askell ... Alex Tamkin Janel Thamkul Jared Kaplan Jack Clark Deep Ganguli 35 207 0 28 Jun 2023
The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographics Matthias Orlikowski Paul Röttger Philipp Cimiano Italy 26 26 0 20 Jun 2023
Intersectionality in Conversational AI Safety: How Bayesian Multilevel Models Help Understand Diverse Perceptions of Safety Christopher Homan Greg Serapio-García Lora Aroyo Mark Díaz Alicia Parrish Vinodkumar Prabhakaran Alex S. Taylor Ding Wang 22 9 0 20 Jun 2023
DICES Dataset: Diversity in Conversational AI Evaluation for Safety Lora Aroyo Alex S. Taylor Mark Díaz Christopher Homan Alicia Parrish Greg Serapio-García Vinodkumar Prabhakaran Ding Wang 26 33 0 20 Jun 2023
When Do Annotator Demographics Matter? Measuring the Influence of Annotator Demographics with the POPQUORN Dataset Jiaxin Pei David Jurgens 34 31 0 12 Jun 2023
Evaluating the Social Impact of Generative AI Systems in Systems and Society Irene Solaiman Zeerak Talat William Agnew Lama Ahmad Dylan K. Baker ... Marie-Therese Png Shubham Singh A. Strait Lukas Struppek Arjun Subramonian ELM EGVM 31 104 0 09 Jun 2023
COBRA Frames: Contextual Reasoning about Effects and Harms of Offensive Statements Xuhui Zhou Haojie Zhu Akhila Yerukola Thomas Davidson Jena D. Hwang Swabha Swayamdipta Maarten Sap 19 33 0 03 Jun 2023