One-sentence Summary

Apple Inc. introduces MKQA, an open-domain question answering benchmark with 10k question-answer pairs aligned across 26 typologically diverse languages (260k pairs total) using language-independent answer representations, and benchmarks generative and extractive models trained on Natural Questions in zero-shot and translation settings, demonstrating that the dataset is challenging even in English, especially for low-resource languages.

Key Contributions

• The paper introduces Multilingual Knowledge Questions and Answers (MKQA), an open-domain QA evaluation set with 10k question-answer pairs aligned across 26 typologically diverse languages, totaling 260k pairs, which provides the widest language coverage for QA evaluation to date.
• This work designs MKQA with language-independent, retrieval-independent answer annotations based on human-answerable questions, enabling fair cross-language comparisons and higher answer coverage (67.58% vs. 38% average in TyDi) than existing resources.
• The paper benchmarks state-of-the-art extractive and generative QA models trained on Natural Questions in zero-shot and translation settings, showing that the dataset is challenging even in English and especially difficult in low-resource languages, indicating ample room for improvement.

Introduction

Question answering systems are increasingly multilingual, but training and evaluation data remain scarce outside high-resource languages like English. Existing multilingual QA datasets are realistic and challenging, yet they suffer from limited linguistic diversity, lack directly comparable examples across languages, and are often tied to passages bundled with the dataset. Extractive-style annotations that depend on specific passages also introduce "translationese" artifacts when translated, and they constrain evaluation to retrieval-based approaches.

The authors introduce Multilingual Knowledge Questions and Answers (MKQA), an open-domain QA evaluation set designed to overcome these limitations. They select 10,000 realistic English queries from Natural Questions and translate them into 25 additional languages and dialects, covering 26 languages across 14 language families. A central design choice is replacing passage-embedded answer spans with high-quality, language- and retrieval-independent annotations linked directly to Wikidata entities and structured value types such as numbers, dates, and strings. This makes MKQA compatible with any QA technique, including knowledge graph, dense and sparse retrieval, and generative approaches, while avoiding translation artifacts. The dataset is fully parallel across all languages, enabling direct and fair comparison of systems across typologically diverse languages. The authors enforce strict annotation quality controls and verify that questions are geographically invariant, so answers remain consistent across cultural contexts. MKQA is the largest and most linguistically diverse open-domain QA evaluation set to date, and baseline results show substantial room for improvement, particularly in lower-resource languages.

Dataset

Dataset Composition and Sources

• The authors introduce MKQA (Multilingual Knowledge Questions and Answers), an evaluation set built for open-domain question answering across multiple languages.
• The source pool is 10,000 queries sampled from Natural Questions (NQ). NQ is chosen because it contains realistic queries generated by information-seeking users.
• The authors aim for three properties in the evaluation set: realistic questions, reliable annotations (verified through inter-annotator agreement), and a flexible task setup that makes few assumptions about the underlying modeling technique, enabling fair comparison across approaches.
• The dataset covers a set of languages selected to maximize typological diversity and world population coverage. The selected languages cover 90.62% of the world population when measured by countries where at least one selected language is official.

Answer Curation Pipeline

The authors build the dataset through a multi-stage answer curation process:

• Raw Answer Collection: 5 annotators independently search the web and either copy or generate an ideal answer for each query. They select an answer type from a defined taxonomy and enter the answer text according to format instructions per type. The taxonomy includes:

  • Atomic value: dates, numbers, and number ranges with or without units.
  • Entities: annotated with Wikidata QIDs, covering generic entities, people, objects, and most locations.
  • Yes/No: binary answers.
  • Short answer: answers that cannot fit into atomic, entity, or binary types but are still short phrases.
  • Long answer: used when no simple factual or short phrase answer exists and a longer or visual explanation is required. These are treated as "Unanswerable" during evaluation for simplicity.
  • Unanswerable: indicates the query is ill-formed or no clear answer exists.

• Answer Resolution: Annotators normalize date and number formats and resolve answer text against Wikidata entities. For short answers, an internal entity linking system generates Wikidata candidate entities. The top 10 suggestions are shown to 3 graders, who select the correct entity or "None of the above." When graders do not agree sufficiently, a domain expert provides the correct reference. This step disambiguates homonyms and collects valid synonyms and aliases.

• Answer Verification: Any normalized answer given by at least 2 of the 5 annotators is admitted as a gold answer. For annotations that fail this agreement threshold, a domain expert reviews all 5 preliminary annotations and makes a final decision, with permission to select, modify, or override existing normalized answers.

• Answer Localization: Verified answers are localized into target languages using two methods. For Wikidata-resolved answers, Wikidata names and aliases in the target language are used. These are transcribed in the native alphabet where appropriate and naturalize unit terms (for example, "November," "century," "acres," "light years"). Date answer templates are generated for every combination of year, month, and day, accommodating American and European date formats and numeric or written-out months. When Wikidata links are not available, professional bilingual translators provide the native equivalent, choosing from transliteration, translation, unchanged, or a mix of these options.

• Query Localization: Bilingual translators translate each query, preserving meaning while keeping natural phrasing. Translators are native speakers of the target language, must pass an entrance exam verifying English fluency, and receive a standard hourly wage. On average, around 16 translators participate per target language for the 10,000 source queries.

Dataset Quality and Statistics

• The authors run a small grading experiment on just over 1% of the data to measure translation quality and answer acceptability (referred to as geographical invariance).
• Query translations are consistently judged acceptable in German, Spanish, and Thai, while Simplified Chinese translations receive lower acceptance. Most translation issues relate to entity localization and domain-specific terms.
• Answer acceptability is highest for English but remains at or above 90% even for languages as linguistically distant from English as Thai.
• The main causes of answer quality issues are:
  • Answers differing based on cultural context (44%), for example, localized versions of an entity having different properties.
  • Generic annotation issues that hold across languages (33%), including time-sensitive answers and ambiguous queries.
  • Entities transliterated incorrectly (11%).
  • Generic translation artifacts (11%) that create mismatches between the question and the language-independent answer.
• Compared to NQ, MKQA has a lower share of "Unanswerable" or "Long" answer types (32.4% versus 63%). This shift reflects a definition change from whether a retrieved passage contains an answer to whether a human with full web access can answer succinctly, so the answer annotations in MKQA reflect only the properties of the question and are independent of any retrieval system.

Metadata and Usage in Evaluation

• The dataset includes supplementary metadata per example: answer type and Wikidata QIDs for entity answers.
• These metadata enable practitioners to perform error analysis with more interpretable metrics and allow answers to be further localized into Wikipedia languages beyond those covered in MKQA.
• The authors encourage future multilingual QA benchmarks to report geographical invariance and benchmark the reliability of examples, including the presence of translation artifacts, as done in this work.

Method

The data collection methodology is structured as a six-step pipeline designed to generate high-quality multilingual question answering data. The process is bifurcated into answer curation and localization phases. Refer to the framework diagram for an overview of the sequential steps.

The pipeline initiates with query sampling, where ten thousand queries are selected from the Natural Questions dataset. These queries serve as the foundation for the answer curation phase. In this phase, five independent annotators research each query to generate or select an ideal answer. The annotators categorize their responses into a predefined taxonomy, which includes atomic values, entities, yes/no answers, short answers, long answers, and unanswerable categories. This categorization allows for automated processing of specific answer types. For instance, atomic values such as dates and numbers with units are formatted to enable automatic linking to Wikidata entities.

Following the collection of raw answers, the system normalizes the formats and resolves entities against the Wikidata knowledge base. This step associates each answer with a unique Wikidata QID, ensuring consistency and disambiguation. The verification phase then ensures the reliability of the answers. If at least two annotators provide the same normalized answer, it is accepted as a gold standard. For cases lacking consensus, a domain expert reviews the annotations and makes a final decision. The expert can select, modify, or override the existing answers to resolve discrepancies.

The localization phase follows, translating the verified English queries and answers into twenty-five target languages. For answer localization, the system primarily leverages Wikidata names and aliases. This approach preserves the native alphabet and naturalizes unit terms while maintaining Arabic numerals. When Wikidata links are unavailable, professional bilingual human translators are employed. These translators are provided with the English query, the answer, and context such as Wikipedia pages. They are instructed to use transliteration, translation, or a combination of both to ensure cultural and linguistic accuracy.

The final step involves query localization. Native speakers of the target languages, who have verified fluency in English, translate the queries. They are instructed to preserve the original meaning while ensuring natural phrasing and using localized entity names where applicable. This rigorous localization process ensures that the dataset is robust and suitable for evaluating multilingual question answering models.

Experiment

The experiments evaluate multilingual open-retrieval QA on MKQA across 26 languages, combining retrieval, machine translation, and extractive or generative readers. Results show that translating queries to English with DPR outperforms multilingual sparse retrieval, and the best system uses Translate Train with XLM-R on gold passages, though all baselines lag well behind English-only benchmarks and struggle most with answerable questions. The analysis reveals that multilingual retrieval is the key bottleneck, while models treat long and truly unanswerable examples similarly, and low-resource languages remain far below the No Answer lower bound.

MKQA provides a multilingual open-domain QA benchmark with 10k realistic questions translated into 26 languages, using Wikidata-linked answers and typed value annotations. The example shows a pistachio production query consistently answered as Iran across languages, with entity and alias annotations for retrieval-independent grading. Each question is human-translated into multiple languages using standardized IETF BCP-47 codes, enabling comparable cross-lingual evaluation. Answers are grounded in Wikidata entities with aliases, supporting flexible grading across QA methods beyond passage-based extraction. Less than 4% of answers vary by geographical or cultural context, ensuring most queries are location-invariant.

The table compares multilingual QA evaluation sets on answer independence, parallel questions, language diversity, and scale. MKQA stands out by offering both answer independence and parallel questions across 26 languages with 260k examples, surpassing other datasets in coverage and flexibility. Existing datasets like MLQA and XQuAD lack answer independence, while XQA and TyDi do not provide parallel questions. MKQA is the only dataset with both answer independence and parallel questions. MKQA covers 26 languages, the most among compared sets, with 260k total examples. Other datasets such as MLQA, XQuAD, and TyDi lack answer independence, while XQA and TyDi do not have parallel questions.

The selected languages are dominated by the Indo-European family, with Germanic and Italic branches represented. English has the highest global reach, followed by Spanish, while several Germanic languages such as Dutch and Scandinavian languages have much smaller speaker shares. This distribution reflects a trade-off between typological diversity and practical population coverage. English is the most widely spoken language in the set, with a reach nearly double that of the next largest language. Germanic languages are the most numerous, but most have very low global reach, highlighting their limited speaker populations. Spanish is the only Romance language listed and is the second most reached language, far ahead of all non-English Germanic languages.

Grading a small sample of the dataset shows that query translations are highly acceptable for German, Spanish, and Thai, while Chinese translations receive lower ratings. Answer quality remains at or above 89% for all tested non-English languages, with English as the baseline performing best. Translation and answer errors often overlap, and answer issues are frequently tied to cultural context differences. German and Spanish query translations reach 99% acceptance, while Chinese lags at about 92%. Answer acceptance stays above 89% even for Thai, a language linguistically distant from English. English answers score about 97% acceptance, slightly higher than all other languages. Incorrect query translations usually make the corresponding answer unacceptable, so errors overlap. About 44% of answer quality issues stem from cultural context, such as differing seasons or voice actors.

Translate-Train with an XLM-R reader using NQ gold passages achieves the best overall performance across retrieval, answerable, and end-to-end metrics. Multilingual sparse retrieval via Elasticsearch underperforms translation-based English DPR, and a naive no-answer baseline sets a lower bound of 32.42% F1. Answerable questions are harder than the aggregate end-to-end metric, and retrieval success remains a key bottleneck. Translate-Train with XLM-R on gold NQ passages yields the highest mean F1, outperforming M-Bert, Translate-Test, and zero-shot approaches. Translation followed by English DPR surpasses multilingual Elasticsearch retrievers, which show weaker retrieval and end-to-end performance. Answerable questions have lower F1 than the combined end-to-end metric, indicating that correctly retrieving the passage is a major challenge.

The evaluation introduces MKQA, a multilingual open-domain QA benchmark with 10k questions translated into 26 languages, uniquely combining answer independence and parallel questions for flexible, retrieval-independent grading. Language selection balances typological diversity with speaker reach, and human evaluation shows high translation acceptance for most languages, though Chinese lags slightly; answer quality remains robust, with cultural context causing some errors. In model experiments, Translate-Train with an XLM-R reader using gold passages achieves the best performance, outperforming multilingual sparse retrieval and zero-shot approaches, while retrieval success remains the primary bottleneck for end-to-end accuracy.