MAKIEVAL 是由慕尼黑大学 MaiNLP 研究实验室联合慕尼黑机器学习中心（MCML）于 2026 年发布的多语言文化知识评估数据集，相关论文成果为 MAKIEVAL: A Multilingual Automatic WiKIdata-based Framework for Cultural Awareness Evaluation for LLMs，旨在为大语言模型提供大规模多语言文化知识评估基准，广泛应用于多语言知识表示与文化知识建模研究。 该数据集包含 7 个大语言模型在 13 种语言、 19 个国家 / 地区、 6 个文化领域下生成的文本及其自动抽取的文化实体与 Wikidata 对齐结果。

数据集构成

• 7 个大语言模型：Llama-3.1-8B-Instruct 、 Llama-3.3-70B-Instruct 、 Mistral-7B-Instruct-v0.1 、 Qwen2.5-7B-Instruct 、 DeepSeek-V3 、 ChatGPT-4o-mini 、 Aya-Expanse-8B
• 13 种语言：阿拉伯语、德语、英语、西班牙语、波斯语、印地语、意大利语、日语、韩语、泰语、土耳其语、简体中文、繁体中文
• 19 个国家 / 地区：阿拉伯联合酋长国、美国、英国、加拿大、澳大利亚、尼日利亚、德国、西班牙、墨西哥、阿根廷、伊朗、印度、意大利、日本、韩国、泰国、土耳其、中国、中国台湾
• 6 个文化领域：食物、饮品、服饰、书籍、音乐、交通

Citation

@inproceedings{zhao-etal-2025-makieval,
title = "{MAKIE}val: A Multilingual Automatic {W}i{K}idata-based Framework for Cultural Awareness Evaluation for {LLM}s",
author = "Zhao, Raoyuan  and
Chen, Beiduo  and
Plank, Barbara  and
Hedderich, Michael A.",
editor = "Christodoulopoulos, Christos  and
Chakraborty, Tanmoy  and
Rose, Carolyn  and
Peng, Violet",
booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
month = nov,
year = "2025",
address = "Suzhou, China",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.findings-emnlp.1256/",
doi = "10.18653/v1/2025.findings-emnlp.1256",
pages = "23104--23136",
ISBN = "979-8-89176-335-7",
abstract = "Large language models (LLMs) are used globally across many languages, but their English-centric pretraining raises concerns about cross-lingual disparities for cultural awareness, often resulting in biased outputs. However, comprehensive multilingual evaluation remains challenging due to limited benchmarks and questionable translation quality. To better assess these disparities, we introduce MAKIEval, an automatic multilingual framework for evaluating cultural awareness in LLMs across languages, regions, and topics. MAKIEval evaluates open-ended text generation, capturing how models express culturally grounded knowledge in natural language. Leveraging Wikidata{'}s multilingual structure as a cross-lingual anchor, it automatically identifies cultural entities in model outputs and links them to structured knowledge, enabling scalable, language-agnostic evaluation without manual annotation or translation. We then introduce four metrics that capture complementary dimensions of cultural awareness: granularity, diversity, cultural specificity, and consensus across languages. We assess 7 LLMs developed from different parts of the world, encompassing both open-source and proprietary systems, across 13 languages, 19 countries and regions, and 6 culturally salient topics (e.g., food, clothing). Notably, we find that models tend to exhibit stronger cultural awareness in English, suggesting that English prompts more effectively activate culturally grounded knowledge. We publicly release our code and data."
}