Uncovering Bias in Large Vision-Language Models at Scale with Counterfactuals

30 May 2024

Phillip Howard

Kathleen C. Fraser

Anahita Bhiwandiwalla

S. Kiritchenko

ArXiv PDF HTML

Papers citing "Uncovering Bias in Large Vision-Language Models at Scale with Counterfactuals"

37 / 37 papers shown

Title
A Stereotype Content Analysis on Color-related Social Bias in Large Vision Language Models Junhyuk Choi Minju Kim Yeseon Hong Bugeun Kim 37 0 0 27 May 2025
Interpreting Social Bias in LVLMs via Information Flow Analysis and Multi-Round Dialogue Evaluation Zhengyang Ji Yifan Jia Shang Gao Yutao Yue 30 0 0 27 May 2025
When Tom Eats Kimchi: Evaluating Cultural Bias of Multimodal Large Language Models in Cultural Mixture Contexts Jun Seong Kim Kyaw Ye Thu Javad Ismayilzada Junyeong Park Eunsu Kim Huzama Ahmad Na Min An James Thorne Alice Oh 58 1 0 21 Mar 2025
LVLM-Compress-Bench: Benchmarking the Broader Impact of Large Vision-Language Model Compression Souvik Kundu Anahita Bhiwandiwalla Sungduk Yu Phillip Howard Tiep Le S. N. Sridhar David Cobbley Hao Kang Vasudev Lal MQ 80 2 0 06 Mar 2025
Revealing and Reducing Gender Biases in Vision and Language Assistants (VLAs) Leander Girrbach Yiran Huang Stephan Alaniz Trevor Darrell Zeynep Akata VLM 102 2 0 25 Oct 2024
Debiasing Large Vision-Language Models by Ablating Protected Attribute Representations Neale Ratzlaff Matthew Lyle Olson Musashi Hinck Shao-Yen Tseng Vasudev Lal Phillip Howard 95 0 0 17 Oct 2024
Lookism: The overlooked bias in computer vision Aditya Gulati Bruno Lepri Nuria Oliver 56 0 0 21 Aug 2024
BiasDora: Exploring Hidden Biased Associations in Vision-Language Models Chahat Raj A. Mukherjee Aylin Caliskan Antonios Anastasopoulos Ziwei Zhu VLM 65 4 0 02 Jul 2024
LLaVA-Gemma: Accelerating Multimodal Foundation Models with a Compact Language Model Musashi Hinck Matthew Lyle Olson David Cobbley Shao-Yen Tseng Vasudev Lal VLM 47 10 0 29 Mar 2024
A Unified Framework and Dataset for Assessing Societal Bias in Vision-Language Models Ashutosh Sathe Prachi Jain Sunayana Sitaram 81 2 0 21 Feb 2024
Examining Gender and Racial Bias in Large Vision-Language Models Using a Novel Dataset of Parallel Images Kathleen C. Fraser S. Kiritchenko 81 38 0 08 Feb 2024
SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examples Phillip Howard Avinash Madasu Tiep Le Gustavo Lujan Moreno Anahita Bhiwandiwalla Vasudev Lal 77 23 0 30 Nov 2023
Improved Baselines with Visual Instruction Tuning Haotian Liu Chunyuan Li Yuheng Li Yong Jae Lee VLM MLLM 118 2,700 0 05 Oct 2023
VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolution S. Hall F. G. Abrantes Hanwen Zhu Grace A. Sodunke Aleksandar Shtedritski Hannah Rose Kirk CoGe 83 44 0 21 Jun 2023
Bias Against 93 Stigmatized Groups in Masked Language Models and Downstream Sentiment Classification Tasks Katelyn Mei Sonia Fereidooni Aylin Caliskan 55 51 0 08 Jun 2023
InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning Wenliang Dai Junnan Li Dongxu Li A. M. H. Tiong Junqi Zhao Weisheng Wang Boyang Albert Li Pascale Fung Steven C. H. Hoi MLLM VLM 107 2,059 0 11 May 2023
On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Research Luiza Amador Pozzobon Beyza Ermis Patrick Lewis Sara Hooker 73 48 0 24 Apr 2023
Visual Instruction Tuning Haotian Liu Chunyuan Li Qingyang Wu Yong Jae Lee SyDa VLM MLLM 529 4,740 0 17 Apr 2023
G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment Yang Liu Dan Iter Yichong Xu Shuohang Wang Ruochen Xu Chenguang Zhu ELM ALM LM&MA 165 1,187 0 29 Mar 2023
DeAR: Debiasing Vision-Language Models with Additive Residuals Ashish Seth Mayur Hemani Chirag Agarwal VLM 30 55 0 18 Mar 2023
MultiModal Bias: Introducing a Framework for Stereotypical Bias Assessment beyond Gender and Race in Vision Language Models Sepehr Janghorbani Gerard de Melo VLM 77 12 0 16 Mar 2023
GPT-4 Technical Report OpenAI OpenAI OpenAI Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad ... Shengjia Zhao Tianhao Zheng Juntang Zhuang William Zhuk Barret Zoph LLMAG MLLM 1.4K 14,313 0 15 Mar 2023
Is ChatGPT a Good NLG Evaluator? A Preliminary Study Jiaan Wang Yunlong Liang Fandong Meng Zengkui Sun Haoxiang Shi Zhixu Li Jinan Xu Jianfeng Qu Jie Zhou LM&MA ELM ALM AI4MH 118 466 0 07 Mar 2023
How well can Text-to-Image Generative Models understand Ethical Natural Language Interventions? Hritik Bansal Da Yin Masoud Monajatipoor Kai-Wei Chang 85 101 0 27 Oct 2022
"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Dataset Eric Michael Smith Melissa Hall Melanie Kambadur Eleonora Presani Adina Williams 101 142 0 18 May 2022
Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics Paula Czarnowska Yogarshi Vyas Kashif Shah 62 109 0 28 Jun 2021
Computer Vision and Conflicting Values: Describing People with Automated Alt Text Margot Hanley Solon Barocas K. Levy Shiri Azenkot Helen Nissenbaum 42 30 0 26 May 2021
Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Models Tejas Srinivasan Yonatan Bisk VLM 59 56 0 18 Apr 2021
Learning Transferable Visual Models From Natural Language Supervision Alec Radford Jong Wook Kim Chris Hallacy Aditya A. Ramesh Gabriel Goh ... Amanda Askell Pamela Mishkin Jack Clark Gretchen Krueger Ilya Sutskever CLIP VLM 921 29,372 0 26 Feb 2021
CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models Nikita Nangia Clara Vania Rasika Bhalerao Samuel R. Bowman 118 679 0 30 Sep 2020
Language (Technology) is Power: A Critical Survey of "Bias" in NLP Su Lin Blodgett Solon Barocas Hal Daumé Hanna M. Wallach 155 1,236 0 28 May 2020
StereoSet: Measuring stereotypical bias in pretrained language models Moin Nadeem Anna Bethke Siva Reddy 99 998 0 20 Apr 2020
Toward Gender-Inclusive Coreference Resolution Yang Trista Cao Hal Daumé 78 143 0 30 Oct 2019
Fairness-Aware Ranking in Search & Recommendation Systems with Application to LinkedIn Talent Search S. Geyik Stuart Ambler K. Kenthapadi 90 382 0 30 Apr 2019
Counterfactual Fairness in Text Classification through Robustness Sahaj Garg Vincent Perot Nicole Limtiaco Ankur Taly Ed H. Chi Alex Beutel 99 260 0 27 Sep 2018
Gender Bias in Coreference Resolution Rachel Rudinger Jason Naradowsky Brian Leonard Benjamin Van Durme 63 641 0 25 Apr 2018
Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods Jieyu Zhao Tianlu Wang Mark Yatskar Vicente Ordonez Kai-Wei Chang 119 933 0 18 Apr 2018