One-sentence Summary

Researchers from Shanghai Artificial Intelligence Laboratory, University of Science and Technology of China, Zhejiang University, and Shanghai Jiao Tong University propose Scalable Visual Pretraining, a paradigm that directly leverages visual documents without text extraction, and demonstrate that it consistently outperforms text-only pretraining for language intelligence across multiple backbones and benchmarks.

Key Contributions

• Converting visually rich documents to plain text causes substantial information loss, and visual pretraining directly on raw document images recovers these discarded visual cues, yielding stronger language intelligence than text-only pretraining on the same underlying corpora.
• An autoregressive visual pretraining framework is introduced that trains a foundation model to predict document patches in latent space, integrating visual representations into the predictive process without requiring text extraction or image-text pairing supervision.
• A unified empirical study with matched data sources across multiple model backbones and scientific reasoning benchmarks establishes visual pretraining as an effective and efficient alternative to text-only pretraining, achieving better performance while using only 25% of the token budget.

Introduction

The authors challenge the assumption that large language models must learn from text alone by showing that scientific documents, with their rich visual layouts, figures, and spatial organization, lose critical structural information when reduced to plain text. Prior work either parses these documents into text for language-only pretraining or treats visual inputs as shallow conditioning context for text prediction, never integrating the visual modality into the predictive objective. The main contribution is Visual Pretraining, an autoregressive framework that learns directly from raw document patches in latent space without any text extraction or image-text pairing, and it consistently outperforms text-only pretraining on matched corpora while using only 25% of the token budget.

Method

The authors introduce VP (Visual Pretraining), a continued pretraining method that augments standard text-based pretraining with a next-visual-latent prediction objective applied directly to raw scientific document pages. The core idea is to treat document images as a sequence of visual patches, retain only the informative foreground regions, and train the language model to autoregressively predict the next visual latent feature. This design aligns visual learning with the causal next-token prediction paradigm of language modeling while drastically reducing the visual context length.

Sparse document representation. Each rendered document page $\mathcal{I}$I is passed through a frozen vision tower $E_{\mathrm{v}}$Ev​, producing a dense grid of visual features $\mathcal{Z} = (z_1, \dots, z_N)$Z=(z1​,…,zN​). Because document pages contain large blank regions (margins, whitespace), the authors compute a foreground mask using simple patch-level statistics such as pixel variance and average luminance. Only non-blank patches are kept, and the retained features are reordered in raster-scan order to form a sparse foreground sequence

This sparse representation preserves the spatial layout of foreground content while dramatically shortening the visual context. Each foreground feature is projected into the LLM hidden space via a learned linear projection, and position indices are reassigned according to the raster order. Multiple foreground sequences can be packed into fixed-length training contexts, with sequence boundaries tracked to prevent cross-sample attention.

Next visual latent prediction. Given the sparse visual sequence, the projected features are fed into the autoregressive LLM backbone under a causal attention mask over visual positions. At each visual position $t$t, the LLM hidden state is mapped back to the frozen visual-latent space by an output projection head, producing a prediction $\hat{\mathbf{z}}_{t+1}$z^t+1​. The training objective is to match this prediction to the true next foreground visual latent $\mathbf{z}_{t+1}$zt+1​ using a contrastive loss with in-batch negatives. For a batch of predicted–target pairs, the softmax probability $p_{ij}$pij​ of matching prediction $i$i to target $j$j is computed from cosine similarities with temperature $\tau$τ. The visual pretraining loss is

All other visual features in the batch serve as negatives, encouraging the model to distinguish the correct next document feature from other patches. This objective mirrors the next-token prediction structure of language modeling but operates on continuous visual latents rather than discrete text tokens.

Joint text and visual pretraining. The final training objective combines the standard text next-token prediction loss with the visual latent prediction loss:

where $\mathcal{L}_{\mathrm{CE}}$LCE​ is the autoregressive cross-entropy loss on text tokens. Text and visual examples are interleaved during training according to a fixed mixing ratio. The LLM, the visual input projection, and the prediction head are updated, while the vision encoder remains frozen.

Training setup and controlled comparison. The authors implement continued pretraining from a fixed checkpoint using the XTuner framework. For both the text-only baseline (TP) and VP, the non-PDF text corpus, optimization recipe, and supervised fine-tuning (SFT) stage are kept identical. The only controlled difference lies in how the additional scientific PDF corpus is represented. In TP, PDF pages are converted to MinerU2.5-parsed text, yielding approximately 80B text tokens. In VP, the same PDF pages are rendered as images and filtered into sparse foreground visual sequences, yielding roughly 20B visual tokens at the main resolution. Consequently, the total CPT token budgets differ (180B for TP vs. 120B for VP), reflecting the more compact representation of the visual stream rather than a different document source. After pretraining, both checkpoints undergo the same SFT recipe and are evaluated under identical zero-shot protocols, ensuring that any performance differences stem from the visual representation itself.

Experiment

The evaluation compares visual pretraining on raw document pages against text pretraining on parsed text from the same scientific PDFs, followed by identical supervised fine-tuning. Across multiple benchmarks and model families, visual pretraining consistently improves scientific reasoning, scales more efficiently with data, and requires only a quarter of the token budget. These gains are driven by visual content such as figures and equations, and the approach also enhances multimodal alignment and visual reasoning without any image–text pair supervision. The overall conclusion is that preserving the native visual form of documents yields richer knowledge extraction and reasoning capabilities.

Visual pretraining on rendered document pages consistently outperforms text pretraining on parsed text from the same scientific PDF corpus across text-only reasoning benchmarks. The gains hold for multimodal and language-only models, and visual pretraining achieves this with substantially lower token usage, indicating that preserving native visual form better captures reasoning-relevant information. Visual pretraining improves GPQA Diamond by up to 3.2 points and MMLU-Pro by up to 2.1 points over the text-pretrained baseline across all tested backbones. On the same document corpus, visual pretraining surpasses text pretraining while using only 25% of the token budget. The advantage is attributed to the visual representation retaining higher-fidelity content from figures, equations, and layouts that text extraction discards or weakens.

Visual pretraining (VP) outperforms both the original base model and text-pretrained (TP) baselines on all evaluated multimodal benchmarks. The gains are largest on visually heavy tasks like ChartQAPro and MathVista, where TP yields negligible improvement or even regresses, showing that unsupervised visual document pretraining transfers effectively to multimodal reasoning without any labeled image–text pairs. VP consistently improves multimodal performance over the base and TP checkpoints across all four benchmarks and both backbone models. The advantage is greatest on visually rich benchmarks, with VP outperforming TP by 5.4 points on ChartQAPro and 4.8 points on MathVista, while TP fails to provide meaningful gains.

Visual pretraining on rendered document pages consistently outperforms text pretraining on parsed text from the same scientific PDF corpus, achieving gains on both text-only reasoning benchmarks (up to 3.2 points on GPQA Diamond and 2.1 on MMLU-Pro) and multimodal tasks, with the largest advantages on visually rich benchmarks like ChartQAPro and MathVista. It achieves these improvements with substantially lower token usage, as the visual representations preserve high-fidelity content from figures, equations, and layouts that text extraction discards or weakens. The benefits transfer effectively to multimodal reasoning without any labeled image-text pairs, while text pretraining often yields negligible or regressive performance on such tasks.