One-sentence Summary

NVIDIA researchers introduce MIRACL-VISION, a multilingual visual document retrieval benchmark extending MIRACL across 18 languages with human-generated questions and a method for removing easy negatives to enable compute-friendly evaluation, revealing that state-of-the-art VLM-based embedding models underperform text-based retrieval by up to $59.7\%$59.7% overall and $12.1\%$12.1% for English, thus highlighting gaps in multilingual visual capabilities.

Key Contributions

• The paper introduces MIRACL-VISION, a multilingual visual document retrieval evaluation benchmark covering 18 languages, built as an extension of the MIRACL dataset and leveraging its human-annotated questions to ensure high-quality retrieval evaluation.
• The paper presents a corpus reduction method that eliminates "easy" negatives from the retrieval pool, yielding a dataset that remains challenging while keeping evaluation computationally manageable.
• Experiments comparing MIRACL-VISION against other benchmarks reveal that state-of-the-art VLM-based embedding models lag behind text-based retrieval models by up to 59.7% across languages and by 12.1% on English-only evaluation, exposing a significant multilingual capability gap in current visual retrieval systems.

Introduction

Retrieval-Augmented Generation (RAG) has become a standard approach for grounding LLMs in domain-specific content, and many companies now build assistants that retrieve from internal documents like reports, contracts, and presentations. Text-based retrieval pipelines require heavy ingestion steps such as OCR for scanned pages, layout detection, and structure-aware chunking, all of which become fragile with complex layouts. A newer alternative represents document pages as images and uses visual language models (VLMs) that combine OCR with embedding capabilities, yielding models such as DSE-Qwen2, GME-Qwen2, ColPali, and ColQwen. However, the benchmarks used to evaluate these visual retrievers, including ViDoRe and VDR, rely on synthetically generated questions, small and non-challenging corpora, and lack multilingual coverage. The authors introduce MIRACL-VISION, a benchmark for multilingual visual document retrieval built on the MIRACL text retrieval dataset, which spans 18 languages including low-resource ones (e.g., Swahili) and non-Latin scripts (e.g., Arabic, Japanese, Korean, Russian). They generate images of the first page of Wikipedia articles from MIRACL queries, with questions originally curated by native speakers, and strategically select hard negatives to keep the corpus computationally manageable. Their evaluation reveals that current state-of-the-art vision embedding models underperform smaller text embedding models by up to 59.7% on text-heavy pages, challenging the assumption that VLM-based retrieval is inherently superior and providing a new resource for the community to track progress on multilingual visual retrieval.

Dataset

The authors introduce MIRACL-VISION, a multilingual visual document retrieval benchmark built by extending the existing MIRACL text retrieval benchmark. Here is a breakdown of how the dataset is composed, processed, and used.

Composition and sources

• MIRACL-VISION is derived from MIRACL, a multilingual text retrieval benchmark covering 18 languages with around 77k human-generated queries.
• MIRACL itself is based on Wikipedia articles, split into chunks for embedding. English Wikipedia alone has 5.7M articles, yielding 32M chunks for the English MIRACL corpus.
• For MIRACL-VISION, the authors replace each ground-truth text passage with an image of the Wikipedia page containing that passage, while reusing the original human-generated questions.

Subset construction and filtering

• Step 1: Only the first paragraph of each Wikipedia article is kept, since the authors could not reliably extract images from arbitrary chunks. This yields the MIRACL-1stParagraph intermediate set, reducing the average corpus from 5.9M chunks to roughly 1M chunks per language.
• Step 2: Questions that no longer have a positive document in the filtered corpus are removed, reducing the average number of queries per language from 750 to 439, producing the MIRACL-1stParagraph set.
• Step 3: The corpus is further reduced using the multilingual-e5-large text embedding model. Questions and documents are embedded, cosine similarity is computed, and only the top-100 (English) or top-50 (other languages) most similar documents per question are kept. This retains only positives and hard negatives, yielding roughly 18,819 documents per language on average in the MIRACL-1stParagraph-Reduced set, which is much smaller but still challenging for retrieval.

Image and text generation

• Step 4: For each remaining document, the authors download the corresponding Wikipedia article, modify the HTML to remove elements like the sidebar and header, and render it with Playwright.
• They extract an image of the first vertical 2048 pixels of the article, then crop it to 980 x 980 pixels and save it to disk.
• A text representation, called MIRACL-VISION-text, is also created by extracting the first 12 sentences from the HTML body as an approximate match to the cropped image content.

Statistics and comparison

• The final MIRACL-VISION contains 18 languages, covering low-resource and non-Latin script languages.
• Query counts per language range from 300 to 483, and the average corpus size per language is about 6 times larger than other visual document retrieval benchmarks.
• Compared to ViDoRe (8 English and 2 French datasets) and VDR-Multilingual (English, French, German, Italian, Spanish), MIRACL-VISION offers much broader language coverage.
• A limitation noted by the authors is that MIRACL-VISION queries are mainly text-based, while ViDoRe and VDR-Multilingual include queries about tables, charts, and infographics. However, the authors argue that text-based queries are still highly relevant for evaluating multilingual capability in vision-based retrieval models.
• Human annotation in MIRACL ensures naturally posed questions, in contrast to the synthetic queries generated by LLMs in ViDoRe and VDR-Multilingual, which tend to repeat keywords from the source document and may not reflect unbiased user queries.

Usage in the paper

• MIRACL-VISION is used as an evaluation benchmark, not for training. The authors compare retrieval performance on MIRACL-VISION against results on ViDoRe and VDR-Multilingual to assess multilingual visual document retrieval capability.
• The reduced corpus size is specifically designed to keep retrieval evaluation tractable while preserving correlated results, since large corpora of a million images per language would be computationally prohibitive.

Method

The authors present a systematic pipeline to extend the text-based MIRACL dataset into MIRACL-VISION, a benchmark for multilingual visual document retrieval. This process is designed to reuse human-generated queries while replacing the ground truth text passages with corresponding document images. The overall generation framework is illustrated in the diagram below.

The methodology proceeds through four distinct stages to transform the source corpus.

First, the pipeline filters the corpus to retain only chunks derived from the first paragraph of each article. Since the original MIRACL corpus consists of Wikipedia articles split into manageable chunks for embedding, extracting images from arbitrary chunks within a long document proved unreliable. By restricting the retrieval target to the first paragraph, the system ensures that the ground truth answer is always located on the first page of the source document. This simplification allows for consistent image extraction.

Second, the system filters the query set to ensure answerability. After restricting the corpus to first-paragraph chunks, certain questions no longer have a valid ground truth document within the filtered set. The authors remove all queries that cannot be answered by the remaining first-paragraph chunks, creating an intermediate dataset named MIRACL-1stParagraph.

Third, the pipeline reduces the corpus size while maintaining evaluation difficulty. A full corpus of first-paragraph chunks remains computationally expensive to process for image extraction and model evaluation. To address this, the authors employ a hard-negative mining strategy. They utilize the multilingual-e5-large text embedding model to embed both the queries and the documents in the corpus. By computing the cosine similarity between query and document embeddings, they identify the top-k most similar documents for each query. The corpus is then pruned to retain only the positive documents and these hard negatives (top-100 for English and top-50 for other languages). This results in a significantly smaller yet challenging dataset, named MIRACL-1stParagraph-Reduced, which preserves the main distractor documents for each query.

Finally, the system generates the visual and text representations. For each document in the reduced corpus, the corresponding Wikipedia article is downloaded. The HTML is modified to render only the main content, removing headers and sidebars. Using the Playwright library, the system captures a screenshot of the first 2048 vertical pixels of the article, which is then cropped to 980x980 pixels to form the image version of the dataset. Concurrently, the text content is extracted from the HTML body, retaining the first 12 sentences as an approximate text representation of the image, creating the MIRACL-VISION-text dataset. An example of this process, showing a user query and the corresponding Wikipedia article image, is provided below.

Experiment

The paper introduces MIRACL-VISION, a new multilingual visual document retrieval benchmark, and compares it against existing benchmarks. Experiments show that MIRACL-VISION is significantly more challenging than existing visual benchmarks, which appear nearly saturated due to their small corpora and synthetic query generation. Text-based embedding models consistently outperform vision-based models on the same content, with particularly large gaps for languages with non-Latin alphabets, while achieving this with far fewer parameters. Additionally, the filtered and reduced text variants derived from MIRACL maintain retrieval accuracy correlation with the original dataset despite a 58x size reduction, though longer extracted HTML text introduces noise that slightly lowers performance.

The table compares query and document counts across the original MIRACL corpus, a version filtered to the first paragraph per article, and MIRACL-VISION, which replaces text ground truth with document images. Filtering reduces both queries and documents, and MIRACL-VISION further cuts document counts substantially while retaining the filtered query set. This reduction creates a smaller but harder retrieval benchmark with retained hard negatives. Filtering to the first paragraph cuts both queries and document chunks across all languages, with document counts dropping by roughly a factor of 3 to 5. MIRACL-VISION keeps the same queries as the filtered version but drastically reduces document counts, often by an order of magnitude or more compared to the filtered corpus. The size reduction in MIRACL-VISION is intended to maintain hard negatives while providing a more challenging visual retrieval task.

MIRACL-VISION is a more extensive and challenging visual document retrieval benchmark, covering 18 languages and 18 datasets with an average of 483 queries and 18,500 documents per dataset. It uses human-generated queries and hard-negative sampling, whereas other benchmarks rely on synthetic queries and random document selection with much smaller corpora. MIRACL-VISION spans 18 languages and 18 datasets, far exceeding the 2 to 5 languages and 5 to 10 datasets of existing benchmarks. Unlike random document selection in other benchmarks, MIRACL-VISION samples hard-negatives from a large corpus, increasing retrieval difficulty. Queries in MIRACL-VISION are human-generated from MIRACL, while other benchmarks use synthetic generation with manual evaluation.

On the original MIRACL benchmark, all evaluated text embedding models achieve an average NDCG@10 of about 0.65, while the derived MIRACL-1stParagraph variants yield markedly higher scores around 0.83, indicating the filtered versions are easier retrieval tasks. The Reduced version, which is 58x smaller, closely matches the full first-paragraph version, confirming that aggressive downsizing preserves relative retrieval quality. Average NDCG@10 rises from 0.6499 on MIRACL to 0.8271 on MIRACL-1stParagraph, showing the variant is substantially easier. MIRACL-1stParagraph-Reduced (0.8306) and MIRACL-1stParagraph (0.8271) scores are close, indicating the 58x size reduction retains hard-negative effectiveness.

Across four VLM-based embedding models, NDCG@10 scores are much lower on MIRACL-VISION than on ViDoRe and vdr-multilingual, with average scores of 0.4795 versus 0.9126 and 0.8712, respectively. The best model, colqwen2-v1.0, reaches near-saturation on the existing benchmarks but drops sharply on MIRACL-VISION, suggesting the latter is a substantially more challenging multilingual visual retrieval benchmark. All models score considerably lower on MIRACL-VISION, with the average NDCG@10 falling to 0.4795 compared to above 0.87 on the other benchmarks. colqwen2-v1.0 achieves the highest scores on vdr-multilingual (0.9604) and ViDoRe (0.8969), but only 0.4728 on MIRACL-VISION. The near-saturated performance on existing benchmarks may stem from their small corpus sizes and synthetic queries that repeat document keywords, while MIRACL-VISION likely reflects real user queries. Among the four models, gme-Qwen2-VL-2B-Inst performs best on MIRACL-VISION with 0.5283, yet still well below the scores on the other benchmarks.

Text embedding models consistently outperform visual embedding models on the MIRACL-VISION benchmark across all languages, with the best text model achieving over 50% higher average NDCG@10 than the best vision model. The gap is smallest for English but widens substantially for languages with non-Latin alphabets. Additionally, a compact text model with far fewer parameters surpasses the larger vision models, highlighting efficiency advantages. Text-based models achieve higher NDCG@10 than visual models for every language, with gaps ranging from 12.1% for English to nearly 60% for Arabic, Hindi, and Thai. The best vision model, gme-Qwen2-VL-2B-Instruct, trails the best text model, bge-m3, by more than 50% on average. A text model with only 305M parameters outperforms vision models that use 1543M parameters, demonstrating a fivefold parameter efficiency advantage.

The experiments show that MIRACL-VISION, a visual document retrieval benchmark built from filtered MIRACL data with human-generated queries and hard negatives, is substantially more challenging than existing visual benchmarks due to its larger scale and realistic query formulation. Across all languages, text-based embedding models consistently outperform visual models by a large margin, with the gap widening for non-Latin scripts, while a compact text model surpasses much larger vision models in efficiency. Furthermore, aggressive downsampling of the corpus preserves retrieval quality, indicating that the reduced benchmark maintains hard negatives and difficulty despite its smaller size.