One-sentence Summary

Researchers from The Chinese University of Hong Kong and Tencent investigate scaling laws for native multimodal pre-training, finding that language and multimodal objectives exhibit distinct compute-optimal behaviors, with data composition strongly influencing multimodal allocation, and they derive an efficiency frontier for model size, token count, and data mixture, while also demonstrating that native pre-training improves pure-text spatial reasoning and multimodal in-context learning via positive cross-modal transfer.

Key Contributions

• Decoupling the joint training loss into separate language and multimodal objectives reveals that language scaling is largely invariant to data composition, while multimodal scaling is highly sensitive to the multimodal data ratio.
• A compute-optimal frontier is derived that explicitly accounts for data mixture, specifying precise allocations of model size and token count as functions of compute budget and data composition.
• Downstream experiments show that native multimodal pre-training induces positive cross-modal transfer, improving pure-text spatial reasoning and enabling robust multimodal in-context learning.

Introduction

While LLMs have strong reasoning, text-only training lacks multimodal grounding. Late-fusion methods alleviate this by coupling a pre-trained language model with a vision encoder, but they introduce a fundamental representation asymmetry because vision and language are learned separately. Native multimodal pre-training from scratch promises deeper integration, yet no scaling laws existed to guide how to allocate a fixed compute budget between model size and data across modalities. The authors fill this gap by deriving compute-optimal scaling laws for native multimodal pre-training, decoupling language and multimodal objectives. They find that language scaling follows composition-invariant allocation rules, while multimodal scaling is highly sensitive to the data mixture, leading to a joint Pareto frontier that prescribes the optimal model size, text token count, and multimodal token count for any compute budget. They also show that native pre-training transfers spatial reasoning to pure-text tasks and enables multimodal in-context learning.

Dataset

The authors construct a training corpus that mixes text and multimodal data:

• Text subset: 250B tokens sourced from web pages, books, academic papers, and similar domains.
• Multimodal subset: 75B tokens produced from web‑crawled image‑text pairs and interleaved image‑text documents. Images are converted into continuous patch embeddings using a single patch embedding layer, turning each image into a sequence of token-like embeddings. No additional vision encoder is used.

The entire dataset (text + multimodal tokens) is used to train a decoder‑only Mixture of Experts (MoE) Transformer. The patch embeddings are fed directly to the model alongside text tokens, treating images as native token sequences. No cropping, filtering, or metadata construction details are provided beyond the standard web‑scale curation.

Method

The authors revisit the fundamental problem of compute-optimal allocation in native multimodal pre-training. Given a fixed computational budget $C$C, the goal is to optimally allocate resources between the model size $N$N (the number of activated non-embedding parameters) and the total number of training tokens $D$D. This trade-off is parameterized by the multimodal data ratio $r$r. The final pre-training loss $L(N, D)$L(N,D) is minimized under the constraint $C = 6ND$C=6ND. Since language and multimodal objectives ($L_{text}$Ltext​ and $L_{mm}$Lmm​) are optimized simultaneously using shared parameters, the authors decouple the allocation problem. They define the effective compute for text and multimodal objectives as $C_{text} = 6ND_{text}$Ctext​=6NDtext​ and $C_{mm} = 6ND_{mm}$Cmm​=6NDmm​, where $D_{text} = D/(1 + r)$Dtext​=D/(1+r) and $D_{mm} = Dr/(1 + r)$Dmm​=Dr/(1+r).

To estimate the compute-optimal allocation, two independent methodologies are employed. The primary estimator is the IsoFLOP profile method. At a predetermined compute budget $C$C, plotting the loss of each model against $\log N$logN generates an IsoFLOP profile, which is modeled as a parabola. The minimum of this parabola identifies the optimal model size $N_{opt}(C)$Nopt​(C), from which the optimal token count follows as $D_{opt}(C) = C / (6N_{opt})$Dopt​(C)=C/(6Nopt​). As shown in the figure below:

The IsoFLOP curves for the language objective reveal stable parabolic minima across various values of $r$r, demonstrating that an optimal model size exists for any given FLOP budget.

The second methodology is the training-curve envelope, which serves as an independent cross-validation mechanism. For a fixed model size $N$N, increasing the token count $D$D produces a trajectory of loss versus total compute $C$C. Pooling the trajectories of all evaluated models and extracting the lower envelope yields the minimum achievable loss for any given budget $C$C. Refer to the figure below:

This diagram illustrates the training curve envelope for the language objective. By extracting the envelope of minimal loss per FLOP from these curves across varying values of $r$r and model sizes, the authors estimate the optimal model size and training token allocation. The compute frontier strictly follows a power law $L(C) = E + (C_c/C)^\beta$L(C)=E+(Cc​/C)β.

Regarding the experimental setup, the model is trained on a mixture of 250B text tokens and 75B multimodal tokens. The architecture employs a Mixture of Experts (MoE) based on a decoder-only Transformer. Instead of traditional vision encoders, a single patch embedding layer projects images directly into continuous patch embeddings. The MoE models are trained using an auxiliary-loss-free approach.

The decoupled analysis reveals distinct scaling behaviors. The compute-optimal allocation for the language objective is highly composition-invariant. The IsoFLOP parametric fits and the training-curve envelope show that the parameter capacity required to minimize $L_{text}$Ltext​ is strictly governed by the isolated budget $C_{text}$Ctext​, demonstrating empirical robustness to the introduction of multimodal tokens. In contrast, the multimodal objective exhibits strictly composition-variant scaling behavior. As $r$r increases, the optimal model size exponent decreases substantially, confirming the inherently data-hungry nature of cross-modal alignment. Processing increasingly dense multimodal data shifts the optimal compute allocation heavily toward data scaling rather than parameter expansion.

To address practical native multimodal pre-training, which must optimize the shared parameter count $N$N under a unified computational budget $C_{total}$Ctotal​, the authors establish a strict Pareto frontier. They employ an asymmetric modeling framework that pairs a composition-invariant language objective with a composition-variant multimodal objective. This asymmetry reflects the structural divergence between the modalities. Text-based language modeling relies on a robust statistical structure, while visual tokens provide external context. Consequently, scaling a unified multimodal foundation model requires a deliberate architectural shift, constraining theoretical parameter expansion in favor of training on substantially larger token budgets to accommodate the data requirements of dense multimodal inputs.

Experiment

The evaluation setup uses 16 text benchmarks and 23 multimodal benchmarks to assess base models under in-context learning. Native multimodal pre-training preserves core language capabilities, with no degradation in aggregate text performance, and produces cross-modal transfer where spatial reasoning from visual data improves purely text-based spatial tasks. Multimodal in-context learning emerges naturally with larger model scale and more training data, with the strongest few-shot gains concentrated on spatial reasoning benchmarks rather than recognition-focused tasks.