Nemotron VLM Dataset v2 is a dataset released by NVIDIA in 2025, containing over 9 million high-quality multimodal samples. It aims to enhance the performance of vision-language models in complex chart analysis, UI understanding, and document processing by introducing video modality, augmenting chain-of-thought reasoning data, and providing an OCR data generation toolchain.

Building upon the 3 million samples in version V1, this dataset significantly expands data scale and diversity, covering various subsets such as multilingual wiki text, visual instruction fine-tuning, document question answering, chart analysis, and video action recognition. Its core feature lies in the introduction of video understanding tasks and the provision of a large volume of Chain-of-Thought (CoT) annotated data containing reasoning processes, helping models break through performance bottlenecks in benchmark tests. The data format is unified as JSONL, including images, videos, audio, and structured text messages. It is compatible with the Megatron Energon multimodal data loader, supporting multilingual document understanding and complex visual reasoning tasks.

Dataset Composition

The dataset contains more than 40 sub-datasets (configs), with all data stored in JSONL format and split into the train set.

• Multilingual Wiki Data: Includes wiki text data in German (wiki_de), English (wiki_en), Spanish (wiki_es), French (wiki_fr), Italian (wiki_it), Japanese (wiki_ja), Korean (wiki_ko), Dutch (wiki_nl), Portuguese (wiki_pt), and Chinese (wiki_zh).
• Visual and Document Understanding: Includes oi_bbox_3 (object detection bounding boxes), tabmwp_cot (math word problem reasoning), sparsetables (sparse tables), mulberry_cot_1/2 (multilingual document reasoning), llava_cot_100k (LLaVA instruction fine-tuning), geomverse_cot (geometry reasoning), mapqa_cot (map question answering), plotqa_cot (plot question answering), visual7w_telling_cot, visual_web_instruct_cot (web instructions), docvqa_cot (document question answering), chartqa_cot (chart question answering), infographicsvqa_cot (infographic question answering), unigeo_cot, nights_cot, mantis_instruct_cot, fintabnet_cot (financial tables), and hiertext (high-resolution text).
• Video Understanding: Includes nextqa (video question answering), clevrer (video reasoning), ego_exo_learn (first-person and third-person perspective learning), kinetics_k710 (action recognition), perception_test_1/2, activity_net_1/2, hacs, hirest_1/2, youcook2_1/2, breakfast_actions (breakfast action recognition), ccpdf_multipage_1/2 (multi-page PDFs), and perception_test_cot (video perception reasoning).