One-sentence Summary

Researchers from The Chinese University of Hong Kong and Tencent demonstrate that native multimodal pre-training from scratch obeys distinct compute laws for language and multimodal losses, with text-heavy data mixtures shifting the compute-optimal allocation toward larger models and defining an efficiency frontier that enables positive cross-modal transfer for spatial reasoning and in-context learning.

Key Contributions

• Compute-optimal scaling laws for native multimodal pre-training are established, demonstrating that minimal loss follows a predictable compute law and optimal model size and token count scale as power laws under a fixed compute budget.
• Decoupling the joint objective into language and multimodal components reveals that the language allocation law is largely invariant to data composition, while the multimodal allocation law is highly sensitive; a joint compute-optimal frontier then specifies model size, token count, and data mixture.
• Downstream evaluation shows that native multimodal pre-training induces positive cross-modal transfer, improving pure-text spatial reasoning and enabling robust multimodal in-context learning.

Introduction

Large language models excel at reasoning and generation, but text-only pre-training lacks grounding in the physical world. The standard late-fusion multimodal approach couples a pre-trained language model with a separate vision encoder, creating an asymmetry where modalities are learned from disjoint distributions under different objectives. Native multimodal pre-training addresses this by training a single model from scratch on interleaved data, yet it raises an open question: how to allocate a fixed compute budget optimally when language and multimodal objectives may conflict. Prior scaling-law research has been limited to unimodal settings or treats the multimodal loss as an aggregate, without disentangling the distinct behavior of each modality. The authors establish compute-optimal scaling laws specifically for native multimodal pre-training. Using IsoFLOP profiles and training-curve envelopes, they show that language scaling is largely insensitive to data composition while multimodal scaling depends heavily on the multimodal data ratio. This divergence yields a compute-optimal Pareto frontier that prescribes the optimal model size, text token count, and multimodal token count for any given compute budget and data mix. Additionally, native multimodal pre-training improves pure-text spatial reasoning and enables robust multimodal in-context learning.

Dataset

The authors construct a training mixture of text-only and multimodal data, totaling roughly 325 billion tokens once images are converted.

• Text corpus (250B tokens)
  • Sources: web pages, books, academic papers, and other general-domain documents.
  • No explicit filtering rules are detailed in the given excerpt, but the text tokens are described as derived from those domains.
• Multimodal corpus (75B tokens equivalent)
  • Sources: large-scale web-crawled image-text pairs and interleaved image-text documents.
  • Processing: each image is directly projected into a sequence of continuous patch embeddings via a single patch embedding layer, producing multimodal tokens. The total is 75B such tokens.
• Usage
  • Both subsets are combined into a single training mixture. The relative ratio is not specified, beyond the token counts indicating roughly 77% text tokens and 23% multimodal tokens.
  • No further cropping strategy or metadata construction is mentioned.

Method

The authors revisit the fundamental problem of compute-optimal allocation in native multimodal pre-training. Given a fixed computational budget $C$C, the goal is to optimally allocate resources between the model size $N$N, defined as the number of activated non-embedding parameters, and the total number of training tokens $D$D. Using the standard approximation $C = 6ND$C=6ND, the objective is to minimize the final pre-training loss $L(N, D)$L(N,D) under this constraint. In native multimodal pre-training, language and multimodal objectives, denoted as $L_{text}$Ltext​ and $L_{mm}$Lmm​, are optimized simultaneously using shared parameters. To investigate whether these objectives follow analogous scaling laws, the authors decouple the allocation problem and analyze each objective independently based on the multimodal data ratio $r$r.

To estimate the compute-optimal allocation, the authors employ two independent methodologies. The primary estimator is the IsoFLOP profile method. At a predetermined compute budget $C$C, plotting the loss of each model against $\log N$logN generates an IsoFLOP profile. A parabola is fitted to each profile, with its minimum identifying the optimal model size $N_{opt}(C)$Nopt​(C). The optimal token count follows algebraically as $D_{opt}(C) = C / (6N_{opt})$Dopt​(C)=C/(6Nopt​). As shown in the figure below, these profiles reveal stable parabolic minima across various values of $r$r.

The second methodology is the training-curve envelope, which serves as an independent cross-validation mechanism. For a fixed model size $N$N, increasing the token count $D$D produces a trajectory of loss versus total compute $C$C. Pooling the trajectories of all evaluated models and extracting the lower envelope yields the minimum achievable loss for any given budget $C$C. As illustrated in the following figure, the compute frontier strictly follows a power law $L(C) = E + (C_c / C)^\beta$L(C)=E+(Cc​/C)β, where $E$E is an irreducible floor, $C_c$Cc​ is a critical compute scale, and $\beta$β is a decay exponent.

The experimental setup utilizes a mixture of text and multimodal datasets, comprising 250B text tokens and 75B multimodal tokens derived from web-crawled image-text pairs. The model architecture employs a Mixture of Experts based on a decoder-only Transformer. Instead of traditional vision encoders, a single patch embedding layer projects images directly into continuous patch embeddings. These models are trained using an auxiliary-loss-free approach.

Analyzing the scaling behavior reveals distinct dynamics for the two objectives. The compute-optimal allocation for the language objective is highly composition-invariant, meaning the parameter capacity required to minimize $L_{text}$Ltext​ is strictly governed by the isolated budget $C_{text}$Ctext​ and remains robust to the introduction of multimodal tokens. In contrast, the multimodal objective exhibits strictly composition-variant scaling behavior. As the multimodal ratio $r$r increases, the optimal model size exponent decreases substantially, confirming the inherently data-hungry nature of cross-modal alignment. Processing dense multimodal data shifts the optimal compute allocation heavily toward data scaling rather than parameter expansion.

To address the joint trade-off under a unified computational budget $C_{total}$Ctotal​, the authors employ an asymmetric modeling framework. This framework pairs a composition-invariant language objective with a composition-variant multimodal objective, reflecting the structural divergence between the modalities. Text-based language modeling relies on a robust statistical structure, whereas visual tokens provide external context. Consequently, scaling a unified multimodal foundation model requires constraining theoretical parameter expansion in favor of training on substantially larger token budgets to accommodate the rapid flattening of the multimodal parameter-scaling curve.

Experiment

Evaluations across 16 text and 23 multimodal benchmarks show that native multimodal pre-training preserves core language capabilities across all scales, while cross-modal transfer significantly improves text-based spatial reasoning as model size increases. Multimodal in-context learning emerges naturally with scaling and sustained training, with few-shot gains concentrated on spatial and diagrammatic tasks rather than fine-grained recognition. Overall, jointly optimizing text and visual objectives yields beneficial synergies without measurable interference in language performance.