One-sentence Summary

DataComp-VLM (DCVLM) is a benchmark for controlled data-centric VLM training experiments that pools 160 datasets across four data types into a $6\text{T}$6T multimodal token corpus, allowing participants to test curation strategies across $1\text{B} \text{--} 8\text{B}$1B–8B models and $6.25\text{B} \text{--} 200\text{B}$6.25B–200B token budgets, where extensive experiments reveal that instruction-heavy data mixing, not filtering, drives superior scaling, as demonstrated by the DCVLM-BASELINE $8\text{B}$8B model achieving $63.6\%$63.6% accuracy on the 33-task core suite after training on $200\text{B}$200B tokens, a $+5.4\text{pp}$+5.4pp gain over the previous best open dataset FINEVISION.

Key Contributions

• DataComp for VLMs (DCVLM) is introduced as a systematic benchmark for data curation in vision-language model pretraining, comprising a 6-trillion-token multimodal corpus from 160 datasets, a 52-benchmark evaluation suite, and infrastructure for scaling experiments across model sizes and token budgets.
• Extensive experiments on DCVLM demonstrate that optimizing data mixture ratios, particularly with instruction-heavy compositions, yields substantial and scale-dependent improvements, whereas applying additional quality filters to already-curated data gives negligible benefits.
• The resulting DCVLM-BASELINE dataset sets a new state of the art for open VLM training data, enabling an 8B model to reach 63.6% accuracy on the core benchmark suite and outperforming the previous best open dataset, FINEVISION, by 5.4 percentage points.

Introduction

The performance of vision-language models (VLMs) is heavily shaped by pretraining data, yet systematic data curation research has lagged behind advances in model architectures and training recipes. While benchmarks like DataComp and DCLM have enabled controlled data-centric experiments in other domains, VLM data curation faces unique hurdles: modern models aggregate many already-curated datasets of different types (image-caption pairs, interleaved documents, text, instruction data), open training data is orders of magnitude smaller than the trillions of tokens used in state-of-the-art systems, the interaction between data type, model scale, and compute budget is vast, and evaluation protocols are inconsistent across studies. The authors introduce DataComp for VLMs (DCVLM), the first benchmark for systematic VLM data curation. It provides a standardized 160-dataset pool totaling 6 trillion multimodal tokens, a scaling ladder from 1B to 8B parameters and 6.25B to 200B training tokens, and a comprehensive 52-benchmark evaluation suite. Through over 1,000 controlled experiments, they demonstrate that optimizing data mixture ratios, especially increasing the proportion of instruction-tuning data, is the dominant lever for improving performance, while applying additional quality filters to already-curated data brings negligible or negative returns. The resulting DCVLM-BASELINE dataset establishes a new state-of-the-art for open VLM training data, and the authors release all artifacts to foster reproducible data curation research.

Dataset

The authors assemble the DCVLM data pool, a large and deliberately heterogeneous collection designed to study VLM data curation strategies. The pool aggregates 160 publicly available datasets, totalling 6 trillion multimodal tokens (tokenized with InternVL-2.5), and is decontaminated against a suite of 52 evaluation benchmarks. It is organized into four data types:

• Image-caption pairs form the largest component and span a wide quality spectrum. High-volume sources like DataComp-1B and ReLAION-2B provide CLIP-score-filtered web-crawled alt-text pairs, while synthetic (ShareGPT-4o) and human-annotated (Pixmo-Cap) sets offer fewer but higher-quality samples.
• Multimodal interleaved documents contain web-scraped image-text sequences from HTML pages, PDFs, and academic papers. Sources include MINT-1T-HTML, MINT-1T-PDF, WanJuan, OmniCorpus, and Multimodal-Textbook. These are the least curated data, relying on minimal URL and heuristic filtering, which results in generally lower quality scores.
• Text-only data helps preserve the language model’s capabilities during multimodal training. It includes instruction and knowledge sources such as FLAN, SlimOrca, Dolly, Numina-Math-1.5, and xCoder80k.
• Multimodal instruction-tuning data consists of single- or multi-turn QA pairs grounded in one or more images (human-written or model-generated). The authors manually categorize this subset into eight capabilities: knowledge, chart & table understanding, general-QA, grounding & counting, math, naive OCR, OCR-QA, and science.

For decontamination, the pool is filtered against the Extended eval suite (52 benchmarks). Multimodal samples with a ResNet-50 SSCD cosine similarity above 0.75 to any test image are removed; text-only samples are filtered with MinHash Jaccard similarity (threshold 0.55).

How the data is used. The pool serves as the fixed source from which participants construct training sets for four competition scales (small, medium, large, x-large). A key design rule keeps the pool-to-training token ratio constant at 30× across all scales: the 6T-token pool always contains 30 times more tokens than the training budget for that scale. Participants apply filtering and mixing strategies to select the final training tokens from the pool. In the baseline experiments, the authors use a mixture of 75% image-caption, 18% text-only, 4% multimodal documents, and 3% instruction-tuning data, derived via length-proportional sampling. The x-large scale corresponds to using the entire pool as the candidate while still enforcing decontamination and curation choices.

Method

The authors leverage data mixing as their primary curation lever to establish a strong baseline for DCVLM, focusing on the allocation of training samples across different data types. Specifically, they optimize the mixture along the ratio of image-caption pairs to instruction-tuning data, while keeping text-only samples and multimodal documents fixed at 15% and 5%, respectively, as supporting components. To determine the optimal allocation, the authors evaluate three distinct mixtures: a Caption-heavy mixture comprising 65% image-caption pairs and 15% instruction-tuning data, a Balanced mixture with 40% image-caption and 40% instruction, and an Instruction-heavy mixture with 10% image-caption and 70% instruction-tuning data. These configurations are rigorously tested across a scaling grid comprising three model sizes (1B, 2B, and 4B) and three token budgets (6.25B, 12.5B, and 25B).

A critical finding from this experimental setup is that data mixing cannot be scale-agnostic. The results reveal a striking interaction between the data mixture and the compute scale. As both the model size and the token budget increase, the Instruction-heavy mix exhibits a markedly steeper scaling slope. It initially performs as the worst mixture at the smallest scale (1B model with 6.25B tokens) but rapidly recovers to become the best performing configuration at the medium (2B model with 25B tokens) and large (4B model with 25B tokens) scales. This crossover pattern demonstrates that mixture rankings established at a small scale do not reliably transfer to larger scales, underscoring the necessity for scale-aware data curation that validates mixture choices across multiple points on the scaling ladder.

Addressing the scalability concerns associated with the Instruction-heavy mix, the authors also investigate the repeatability of instruction-tuning data. Since instruction-tuning datasets are typically much smaller than web-crawled image-caption pairs, a 70% allocation might necessitate extreme data repetitions. To evaluate this, they hold all non-instruction data sources fixed and randomly subsample the instruction-tuning data to induce up to 2x, 4x, and 8x repetitions at the medium scale. The performance degrades gracefully under these conditions, with each doubling of the repetition factor costing roughly 0.5% to 1.0% in performance. Notably, the Instruction-heavy mix with 2x repetitions still matches the Caption-heavy mix with fully unique data, and at 4x repetitions, it remains superior to the base mix. This indicates that the benefits of an optimal mixture outweigh the costs of moderate data repetition.

Experiment

The DCVLM benchmark provides a controlled framework for studying VLM data curation across multiple compute scales, with a heterogeneous pool of pre-filtered web and instruction data. Extensive filtering experiments reveal that quality-based filtering yields negligible improvements on already-curated data and that implicit mixture changes from global filtering can drive performance differences rather than sample quality alone. Control studies confirm that pretraining data choices transfer robustly to supervised fine-tuning and across language model initializations, and a simple instruction-heavy mixture baseline outperforms prior open pretraining datasets, with gains that grow at larger scales.

The DCVLM benchmark defines four training scales—small, medium, large, and x-large—where each step increases total compute eightfold by doubling model size and quadrupling token budget, while keeping the curation pool 30 times the training tokens. Across all scales, the DCVLM-BASELINE recipe consistently outperforms prior open pretraining datasets such as FINEVISION, with performance margins that grow larger as scale increases. Notably, a model trained at the large scale on this recipe can match or exceed a larger model trained on FINEVISION at the x-large scale. Each successive scale represents an 8× compute increase: model parameters double and training tokens quadruple, starting from 1B parameters and 6.25B tokens at small scale to 8B parameters and 200B tokens at x-large. The pool of available data for curation is fixed at 30 times the training token budget across all scales, from 187.5B tokens at small scale to 6T at x-large. DCVLM-BASELINE shows progressive gains over FINEVISION on the 33-task core evaluation, widening from +0.3pp at small scale to +5.4pp at x-large scale. A 4B model trained on DCVLM-BASELINE for 100B tokens (large scale) outperforms an 8B model trained on FINEVISION for 200B tokens (x-large scale), demonstrating superior data efficiency.

Instruction-heavy data mixtures degrade gracefully when instruction-tuning samples are repeated, losing about 0.5 to 1.0 percentage points in average core performance per doubling of repeats. Even with twofold repetitions, the instruction-heavy mix matches the caption-heavy mix that uses fully unique data, and it remains competitive at higher repetition factors. The instruction-heavy mixture with 2× repetitions achieves an average core score of 50.2%, matching the 50.3% of the caption-heavy mix with unique data. Each doubling of the repetition factor reduces performance by roughly 0.5–1.0 percentage points, indicating mild sensitivity to data reuse.

A simple, instruction-heavy data mixture that forgoes filtering yields consistent improvements over existing open VLM pretraining datasets. The advantage over the previous best dataset, FineVision, increases from 0.3 points at the smallest scale to 5.4 points at the largest scale on core benchmarks, and a 4B model trained on this recipe matches or exceeds an 8B FineVision model. DCVLM-Baseline outperforms FineVision across all scales, with gains growing from +0.3 points at the small scale to +5.4 points at the x-large scale on the core evaluation suite. A 4B model trained on DCVLM-Baseline for 100B tokens surpasses an 8B model trained on FineVision for 200B tokens on the extended 52-task benchmark.

The DCVLM-BASELINE curation recipe, a simple instruction-heavy mixture without filtering, consistently outperforms prior open pretraining datasets such as FINEVISION, with performance advantages that grow from small to x‑large scale. This data efficiency is so pronounced that a 4B‑parameter model trained at the large scale matches or exceeds an 8B model trained on FINEVISION at the x‑large scale. Moreover, the recipe degrades gracefully under data repetition, losing only 0.5–1.0 points per doubling of repeated instruction‑tuning samples, and remains competitive even with higher repetition factors.