The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarks

18 October 2022

Daniel Khashabi

Papers citing "The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarks"

28 / 28 papers shown

Title
Agree to Disagree? A Meta-Evaluation of LLM Misgendering Arjun Subramonian Vagrant Gautam Preethi Seshadri Dietrich Klakow Kai-Wei Chang Yizhou Sun 27 1 0 23 Apr 2025
How far can bias go? -- Tracing bias from pretraining data to alignment Marion Thaler Abdullatif Köksal Alina Leidinger Anna Korhonen Hinrich Schutze 69 0 0 28 Nov 2024
WinoPron: Revisiting English Winogender Schemas for Consistency, Coverage, and Grammatical Case Vagrant Gautam Julius Steuer Eileen Bingert Ray Johns Anne Lauscher Dietrich Klakow 48 3 0 09 Sep 2024
Perceptions of Linguistic Uncertainty by Language Models and Humans Catarina G Belém Markelle Kelly M. Steyvers Sameer Singh Padhraic Smyth 43 3 0 22 Jul 2024
When in Doubt, Cascade: Towards Building Efficient and Capable Guardrails Manish Nagireddy Inkit Padhi Soumya Ghosh P. Sattigeri 43 1 0 08 Jul 2024
Exploring Subjectivity for more Human-Centric Assessment of Social Biases in Large Language Models Paula Akemi Aoyagui Sharon Ferguson Anastasia Kuzminykh 50 0 0 17 May 2024
Are Models Biased on Text without Gender-related Language? Catarina G Belém P. Seshadri Yasaman Razeghi Sameer Singh 36 8 0 01 May 2024
Detectors for Safe and Reliable LLMs: Implementations, Uses, and Limitations Swapnaja Achintalwar Adriana Alvarado Garcia Ateret Anaby-Tavor Ioana Baldini Sara E. Berger ... Aashka Trivedi Kush R. Varshney Dennis L. Wei Shalisha Witherspooon Marcel Zalmanovici 33 10 0 09 Mar 2024
KorNAT: LLM Alignment Benchmark for Korean Social Values and Common Knowledge Jiyoung Lee Minwoo Kim Seungho Kim Junghwan Kim Seunghyun Won Hwaran Lee Edward Choi ALM 29 11 0 21 Feb 2024
ViSAGe: A Global-Scale Analysis of Visual Stereotypes in Text-to-Image Generation Akshita Jha Vinodkumar Prabhakaran Remi Denton Sarah Laszlo Shachi Dave Rida Qadri Chandan K. Reddy Sunipa Dev 42 12 0 12 Jan 2024
SocialStigmaQA: A Benchmark to Uncover Stigma Amplification in Generative Language Models Manish Nagireddy Lamogha Chiazor Moninder Singh Ioana Baldini 19 17 0 12 Dec 2023
Tackling Bias in Pre-trained Language Models: Current Trends and Under-represented Societies Vithya Yogarajan Gillian Dobbie Te Taka Keegan R. Neuwirth ALM 43 11 0 03 Dec 2023
Women Are Beautiful, Men Are Leaders: Gender Stereotypes in Machine Translation and Language Modeling Matúš Pikuliak Andrea Hrckova Stefan Oresko Marián Simko 26 5 0 30 Nov 2023
Bias and Fairness in Large Language Models: A Survey Isabel O. Gallegos Ryan A. Rossi Joe Barrow Md Mehrab Tanjim Sungchul Kim Franck Dernoncourt Tong Yu Ruiyi Zhang Nesreen Ahmed AILaw 19 486 0 02 Sep 2023
CALM : A Multi-task Benchmark for Comprehensive Assessment of Language Model Bias Vipul Gupta Pranav Narayanan Venkit Hugo Laurenccon Shomir Wilson R. Passonneau 46 12 0 24 Aug 2023
The Bias Amplification Paradox in Text-to-Image Generation P. Seshadri Sameer Singh Yanai Elazar DiffM 16 39 0 01 Aug 2023
Sociodemographic Bias in Language Models: A Survey and Forward Path Vipul Gupta Pranav Narayanan Venkit Shomir Wilson R. Passonneau 42 20 0 13 Jun 2023
MISGENDERED: Limits of Large Language Models in Understanding Pronouns Tamanna Hossain Sunipa Dev Sameer Singh AILaw 25 34 0 06 Jun 2023
Keeping Up with the Language Models: Robustness-Bias Interplay in NLI Data and Models Ioana Baldini Chhavi Yadav Payel Das Kush R. Varshney MLAU 32 3 0 22 May 2023
It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performance Arjun Subramonian Xingdi Yuan Hal Daumé Su Lin Blodgett 37 17 0 15 May 2023
Challenges to Evaluating the Generalization of Coreference Resolution Models: A Measurement Modeling Perspective Ian Porada Alexandra Olteanu Kaheer Suleman Adam Trischler Jackie C.K. Cheung 32 6 0 16 Mar 2023
BiasTestGPT: Using ChatGPT for Social Bias Testing of Language Models Rafal Kocielnik Shrimai Prabhumoye Vivian Zhang Roy Jiang R. Alvarez Anima Anandkumar 36 6 0 14 Feb 2023
Scaling Instruction-Finetuned Language Models Hyung Won Chung Le Hou Shayne Longpre Barret Zoph Yi Tay ... Jacob Devlin Adam Roberts Denny Zhou Quoc V. Le Jason W. Wei ReLM LRM 62 2,989 0 20 Oct 2022
Quantifying Social Biases Using Templates is Unreliable P. Seshadri Pouya Pezeshkpour Sameer Singh 51 33 0 09 Oct 2022
BBQ: A Hand-Built Bias Benchmark for Question Answering Alicia Parrish Angelica Chen Nikita Nangia Vishakh Padmakumar Jason Phang Jana Thompson Phu Mon Htut Sam Bowman 217 367 0 15 Oct 2021
On Generalization in Coreference Resolution Shubham Toshniwal Patrick Xia Sam Wiseman Karen Livescu Kevin Gimpel 118 36 0 20 Sep 2021
The Woman Worked as a Babysitter: On Biases in Language Generation Emily Sheng Kai-Wei Chang Premkumar Natarajan Nanyun Peng 211 616 0 03 Sep 2019
A Decomposable Attention Model for Natural Language Inference Ankur P. Parikh Oscar Täckström Dipanjan Das Jakob Uszkoreit 201 1,367 0 06 Jun 2016