MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMs

MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMs

27 May 2025

Michael A. Hedderich

Papers citing "MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMs"

19 / 19 papers shown

Title
BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages Junho Myung Nayeon Lee Yi Zhou Jiho Jin Rifki Afina Putri ... Seid Muhie Yimam Mohammad Taher Pilehvar N. Ousidhoum Jose Camacho-Collados Alice Oh 135 50 0 17 Jan 2025
Survey of Cultural Awareness in Language Models: Text and Beyond Siddhesh Pawar Junyeong Park Jiho Jin Arnav Arora Junho Myung Srishti Yadav Faiz Ghifari Haznitrama Inhwa Song Alice Oh Isabelle Augenstein 47 31 0 30 Oct 2024
JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding Benchmark for Culture-aware Evaluation Shota Onohara Atsuyuki Miyai Yuki Imajuku Kazuki Egashira Jeonghun Baek Xiang Yue Graham Neubig Kiyoharu Aizawa OSLM 153 6 0 22 Oct 2024
CulturalBench: A Robust, Diverse, and Challenging Cultural Benchmark by Human-AI CulturalTeaming Yu Ying Chiu Liwei Jiang Bill Yuchen Lin Chan Young Park Shuyue Stella Li ... Mehar Bhatia Maria Antoniak Yulia Tsvetkov Vered Shwartz Yejin Choi ELM ALM 83 0 0 03 Oct 2024
Cross-Lingual and Cross-Cultural Variation in Image Descriptions Uri Berger Edoardo M. Ponti 46 1 0 25 Sep 2024
Benchmarking Vision Language Models for Cultural Understanding Shravan Nayak Kanishk Jain Rabiul Awal Siva Reddy Sjoerd van Steenkiste Lisa Anne Hendricks Karolina Stañczak Aishwarya Agrawal VLM CoGe 105 36 0 15 Jul 2024
M5 -- A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Tasks Florian Schneider Sunayana Sitaram VLM 59 11 0 04 Jul 2024
Extrinsic Evaluation of Cultural Competence in Large Language Models Shaily Bhatt Fernando Diaz ELM EGVM 77 9 0 17 Jun 2024
Ethical Reasoning and Moral Value Alignment of LLMs Depend on the Language we Prompt them in Utkarsh Agarwal Kumar Tanmay Aditi Khandelwal Monojit Choudhury LRM 53 16 0 29 Apr 2024
CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologies Weiyan Shi Ryan Li Yutong Zhang Caleb Ziems Chunhua yu R. Horesh Rogerio Abreu de Paula Diyi Yang 74 34 0 23 Apr 2024
Measuring Political Bias in Large Language Models: What Is Said and How It Is Said Yejin Bang Delong Chen Nayeon Lee Pascale Fung 61 39 0 27 Mar 2024
Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model Ahmet Üstün Viraat Aryabumi Zheng-Xin Yong Wei-Yin Ko Daniel D'souza ... Shayne Longpre Niklas Muennighoff Marzieh Fadaee Julia Kreutzer Sara Hooker ALM ELM SyDa LRM 63 221 0 12 Feb 2024
CPopQA: Ranking Cultural Concept Popularity by LLMs Ming Jiang Mansi Joshi 57 7 0 14 Nov 2023
"Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letters Yixin Wan George Pu Jiao Sun Aparna Garimella Kai-Wei Chang Nanyun Peng 84 186 0 13 Oct 2023
KoBBQ: Korean Bias Benchmark for Question Answering Jiho Jin Jiseon Kim Nayeon Lee Haneul Yoo Alice Oh Hwaran Lee 63 33 0 31 Jul 2023
Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models Myra Cheng Esin Durmus Dan Jurafsky 47 196 0 29 May 2023
Toxicity in ChatGPT: Analyzing Persona-assigned Language Models Ameet Deshpande Vishvak Murahari Tanmay Rajpurohit Ashwin Kalyan Karthik Narasimhan LM&MA LLMAG 62 360 0 11 Apr 2023
Language (Technology) is Power: A Critical Survey of "Bias" in NLP Su Lin Blodgett Solon Barocas Hal Daumé Hanna M. Wallach 155 1,234 0 28 May 2020
Social Biases in NLP Models as Barriers for Persons with Disabilities Ben Hutchinson Vinodkumar Prabhakaran Emily L. Denton Kellie Webster Yu Zhong Stephen Denuyl 57 313 0 02 May 2020