Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotation

1 May 2022

Papers citing "Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotation"

50 / 54 papers shown

Title
Empirically evaluating commonsense intelligence in large language models with large-scale human judgments Tuan Dung Nguyen Duncan J. Watts Mark E. Whiting ELM 24 0 0 15 May 2025
Redefining Toxicity: An Objective and Context-Aware Approach for Stress-Level-Based Detection Sergey Berezin R. Farahbakhsh Noel Crespi 53 0 0 20 Mar 2025
Validating LLM-as-a-Judge Systems in the Absence of Gold Labels Luke M. Guerdan Solon Barocas Kenneth Holstein Hanna M. Wallach Zhiwei Steven Wu Alexandra Chouldechova ALM ELM 209 0 0 13 Mar 2025
Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptions Matthias Orlikowski Jiaxin Pei Paul Röttger Philipp Cimiano David Jurgens Dirk Hovy 59 1 0 28 Feb 2025
Hope vs. Hate: Understanding User Interactions with LGBTQ+ News Content in Mainstream US News Media through the Lens of Hope Speech Jonathan Pofcher Christopher Homan Randall Sell Ashiqur R. KhudaBukhsh 96 0 0 13 Feb 2025
Mitigating Trauma in Qualitative Research Infrastructure: Roles for Machine Assistance and Trauma-Informed Design Emily Tseng Thomas Ristenpart Nicola Dell 80 1 0 22 Dec 2024
Perceiving and Countering Hate: The Role of Identity in Online Responses Kaike Ping James Hawdon Eugenia H. Rho 37 0 0 03 Nov 2024
Re-examining Sexism and Misogyny Classification with Annotator Attitudes Aiqi Jiang Nikolas Vitsakis Tanvi Dinkar Gavin Abercrombie Ioannis Konstas 42 1 0 04 Oct 2024
Assessing the Level of Toxicity Against Distinct Groups in Bangla Social Media Comments: A Comprehensive Investigation Mukaffi Bin Moin Pronay Debnath Usafa Akther Rifa Rijeet Bin Anis 31 0 0 25 Sep 2024
Identity-related Speech Suppression in Generative AI Content Moderation Oghenefejiro Isaacs Anigboro Charlie M. Crawford Danaë Metaxa Sorelle A. Friedler Sorelle A. Friedler 21 0 0 09 Sep 2024
Rater Cohesion and Quality from a Vicarious Perspective Deepak Pandita Tharindu Cyril Weerasooriya Sujan Dutta Sarah K. K. Luger Tharindu Ranasinghe Ashiqur R. KhudaBukhsh Marcos Zampieri Christopher M. Homan 33 1 0 15 Aug 2024
Ontology of Belief Diversity: A Community-Based Epistemological Approach Tyler Fischella Erin van Liemt Qiuyi Qiuyi Zhang 27 0 0 25 Jul 2024
STAR: SocioTechnical Approach to Red Teaming Language Models Laura Weidinger John F. J. Mellor Bernat Guillen Pegueroles Nahema Marchal Ravin Kumar ... Mark Diaz Stevie Bergman Mikel Rodriguez Verena Rieser William S. Isaac VLM 39 7 0 17 Jun 2024
LLM-Mediated Domain-Specific Voice Agents: The Case of TextileBot Shu Zhong Elia Gatti James Hardwick Miriam Ribul Youngjun Cho Marianna Obrist 43 3 0 15 Jun 2024
Whose Preferences? Differences in Fairness Preferences and Their Impact on the Fairness of AI Utilizing Human Feedback Emilia Agis Lerner Florian E. Dorner Elliott Ash Naman Goel 38 1 0 09 Jun 2024
Exploring Human-AI Perception Alignment in Sensory Experiences: Do LLMs Understand Textile Hand? Shu Zhong Elia Gatti Youngjun Cho Marianna Obrist 49 3 0 05 Jun 2024
Safeguarding Large Language Models: A Survey Yi Dong Ronghui Mu Yanghao Zhang Siqi Sun Tianle Zhang ... Yi Qi Jinwei Hu Jie Meng Saddek Bensalem Xiaowei Huang OffRL KELM AILaw 35 17 0 03 Jun 2024
The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels Eve Fleisig Su Lin Blodgett Dan Klein Zeerak Talat 27 13 0 09 May 2024
Towards Geographic Inclusion in the Evaluation of Text-to-Image Models Melissa Hall Samuel J. Bell Candace Ross Adina Williams M. Drozdzal Adriana Romero Soriano EGVM 33 4 0 07 May 2024
D3CODE: Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluation Aida Mostafazadeh Davani Mark Díaz Dylan K. Baker Vinodkumar Prabhakaran 34 8 0 16 Apr 2024
NLP Systems That Can't Tell Use from Mention Censor Counterspeech, but Teaching the Distinction Helps Kristina Gligorić Myra Cheng Lucia Zheng Esin Durmus Dan Jurafsky 42 9 0 02 Apr 2024
Understanding Subjectivity through the Lens of Motivational Context in Model-Generated Image Satisfaction Senjuti Dutta Sherol Chen Sunny Mak Amnah Ahmad Katherine M. Collins Alena Butryna Deepak Ramachandran Krishnamurthy Dvijotham Ellie Pavlick Ravi Rajakumar EGVM 24 1 0 27 Feb 2024
Wikibench: Community-Driven Data Curation for AI Evaluation on Wikipedia Tzu-Sheng Kuo Aaron L Halfaker Zirui Cheng Jiwoo Kim Meng-Hsin Wu Tongshuang Wu Kenneth Holstein Haiyi Zhu 62 21 0 21 Feb 2024
A Scoping Study of Evaluation Practices for Responsible AI Tools: Steps Towards Effectiveness Evaluations G. Berman Nitesh Goyal Michael A. Madaio ELM 42 20 0 30 Jan 2024
Generalized People Diversity: Learning a Human Perception-Aligned Diversity Representation for People Images Hansa Srinivasan Candice Schumann Aradhana Sinha David Madras Gbolahan O. Olanubi Alex Beutel Susanna Ricco Jilin Chen 27 5 0 25 Jan 2024
Disentangling Perceptions of Offensiveness: Cultural and Moral Correlates Aida Mostafazadeh Davani Mark Díaz Dylan K. Baker Vinodkumar Prabhakaran AAML 23 14 0 11 Dec 2023
A Taxonomy of Rater Disagreements: Surveying Challenges & Opportunities from the Perspective of Annotating Online Toxicity Wenbo Zhang Hangzhi Guo Ian D Kivlichan Vinodkumar Prabhakaran Davis Yadav Amulya Yadav 23 2 0 07 Nov 2023
Modeling subjectivity (by Mimicking Annotator Annotation) in toxic comment identification across diverse communities Senjuti Dutta Sid Mittal Sherol Chen Deepak Ramachandran Ravi Rajakumar Ian D Kivlichan Sunny Mak Alena Butryna Praveen Paritosh University of Tennessee 39 5 0 01 Nov 2023
Getting aligned on representational alignment Ilia Sucholutsky Lukas Muttenthaler Adrian Weller Andi Peng Andreea Bobu ... Thomas Unterthiner Andrew Kyle Lampinen Klaus-Robert Muller M. Toneva Thomas L. Griffiths 61 74 0 18 Oct 2023
Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations Zhuoyan Li Hangxiao Zhu Zhuoran Lu Ming Yin SyDa 69 67 0 11 Oct 2023
On the definition of toxicity in NLP Sergey Berezin R. Farahbakhsh Noel Crespi 21 0 0 03 Oct 2023
Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Prompting Tilman Beck Hendrik Schuff Anne Lauscher Iryna Gurevych 35 32 0 13 Sep 2023
How Crowd Worker Factors Influence Subjective Annotations: A Study of Tagging Misogynistic Hate Speech in Tweets Danula Hettiachchi I. Holcombe-James Stephanie Livingstone Anjalee de Silva Matthew Lease Flora D. Salim Mark Sanderson 18 10 0 03 Sep 2023
Exploring Cross-Cultural Differences in English Hate Speech Annotations: From Dataset Construction to Analysis Nayeon Lee Chani Jung Jun-Hee Myung Jiho Jin Jose Camacho-Collados Juho Kim Alice H. Oh 44 14 0 31 Aug 2023
`It is currently hodgepodge'': Examining AI/ML Practitioners' Challenges during Co-production of Responsible AI Values R. Varanasi Nitesh Goyal 31 46 0 14 Jul 2023
Leveraging Contextual Counterfactuals Toward Belief Calibration Qiuyi Zhang Zhang Michael S. Lee Sherol Chen 29 1 0 13 Jul 2023
Towards Measuring the Representation of Subjective Global Opinions in Language Models Esin Durmus Karina Nyugen Thomas I. Liao Nicholas Schiefer Amanda Askell ... Alex Tamkin Janel Thamkul Jared Kaplan Jack Clark Deep Ganguli 35 207 0 28 Jun 2023
The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographics Matthias Orlikowski Paul Röttger Philipp Cimiano Italy 26 26 0 20 Jun 2023
DICES Dataset: Diversity in Conversational AI Evaluation for Safety Lora Aroyo Alex S. Taylor Mark Díaz Christopher Homan Alicia Parrish Greg Serapio-García Vinodkumar Prabhakaran Ding Wang 26 33 0 20 Jun 2023
Designing Closed-Loop Models for Task Allocation Vijay Keswani L. E. Celis K. Kenthapadi Matthew Lease 16 0 0 31 May 2023
Hate Raids on Twitch: Understanding Real-Time Human-Bot Coordinated Attacks in Live Streaming Communities Jie Cai S. Chowdhury Hongyang Zhou D. Y. Wohn 22 10 0 25 May 2023
AI's Regimes of Representation: A Community-centered Study of Text-to-Image Models in South Asia Rida Qadri Renee Shelby Cynthia L. Bennett Emily Denton 24 67 0 19 May 2023
PaLM 2 Technical Report Rohan Anil Andrew M. Dai Orhan Firat Melvin Johnson Dmitry Lepikhin ... Ce Zheng Wei Zhou Denny Zhou Slav Petrov Yonghui Wu ReLM LRM 86 1,147 0 17 May 2023
Consensus and Subjectivity of Skin Tone Annotation for ML Fairness Candice Schumann Gbolahan O. Olanubi Auriel Wright Ellis P. Monk Courtney Heldreth Susanna Ricco 30 21 0 16 May 2023
On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Research Luiza Amador Pozzobon B. Ermiş Patrick Lewis Sara Hooker 38 45 0 24 Apr 2023
Whose Opinions Do Language Models Reflect? Shibani Santurkar Esin Durmus Faisal Ladhak Cinoo Lee Percy Liang Tatsunori Hashimoto 19 383 0 30 Mar 2023
Investigating Labeler Bias in Face Annotation for Machine Learning Luke Haliburton Sinksar Ghebremedhin Robin Welsch Albrecht Schmidt Sven Mayer 24 4 0 24 Jan 2023
The Reasonable Effectiveness of Diverse Evaluation Data Lora Aroyo Mark Díaz Christopher Homan Vinodkumar Prabhakaran Alex S. Taylor Ding Wang 24 9 0 23 Jan 2023
Scaling Instruction-Finetuned Language Models Hyung Won Chung Le Hou Shayne Longpre Barret Zoph Yi Tay ... Jacob Devlin Adam Roberts Denny Zhou Quoc V. Le Jason W. Wei ReLM LRM 62 2,989 0 20 Oct 2022
A Keyword Based Approach to Understanding the Overpenalization of Marginalized Groups by English Marginal Abuse Models on Twitter Kyra Yee Alice Schoenauer Sebag Olivia Redfield Emily Sheng Matthias Eck Luca Belli 22 2 0 07 Oct 2022