One-sentence Summary

Apple introduces MobileCLIP, a family of efficient image-text models whose novel multi-modal reinforced training transfers knowledge from a captioning model and an ensemble of CLIP encoders via a reinforced dataset to avoid train-time compute overhead, with MobileCLIP-S2 delivering $2.3\times$2.3× faster inference than $\text{ViT-B/16}$ViT-B/16 CLIP while achieving higher accuracy, new state-of-the-art latency-accuracy tradeoffs, and $10\times\text{–}1000\times$10×–1000× improved learning efficiency across 38 benchmarks.

Key Contributions

• Introduces MobileCLIP, a family of efficient convolution-transformer hybrid image and text encoders optimized for low-latency, on-device inference. The MobileCLIP-S2 variant runs 2.3x faster while achieving higher accuracy than the previous best ViT-B/16 based CLIP model, setting a new state-of-the-art latency-accuracy tradeoff for zero-shot classification and retrieval.
• Proposes multi-modal reinforced training, a strategy that transfers knowledge from a pre-trained image captioning model and an ensemble of strong CLIP encoders into offline stored targets, thereby avoiding train-time compute overhead. This approach improves a ViT-B/16 based CLIP model by +2.9% average performance across 38 evaluation benchmarks compared to the previous best.
• Contributes DataCompDR-12M and DataCompDR-1B, reinforced variants of the DataComp dataset enriched with synthetic captions and ensemble-derived targets. Training on these datasets demonstrates 10x-1000x learning efficiency gains relative to non-reinforced CLIP training.

Introduction

Large image-text models like CLIP achieve strong zero-shot performance but are too large and slow for mobile deployment. Prior work faced a critical tradeoff: training efficient architectures at scale is computationally prohibitive, while small-scale training yields poor accuracy that does not guide design choices. Additionally, smaller models inherently lose accuracy, requiring better training methods to compensate.

The authors introduce a dataset reinforcement strategy that enriches the DataComp dataset with synthetic captions and embeddings from an ensemble of strong pretrained CLIP models, resulting in DataCompDR. This approach dramatically improves learning efficiency, enabling rapid architecture exploration. Using it, they design MobileCLIP, a family of hybrid CNN-transformer encoders with structural reparametrization, achieving state-of-the-art latency-accuracy tradeoffs, with the fastest variant being 5 times faster and 3 times smaller than standard ViT-B/16 CLIP while maintaining similar accuracy.

Dataset

The authors introduce a dataset reinforcement process to improve the quality of web-sourced image-text data for CLIP-style training. Below is a breakdown of the dataset composition, processing, and usage.

Dataset composition and sources

• The base data comes from large-scale web-sourced image-text pairs, which are inherently noisy.
• To reduce noise, the authors leverage existing filtering mechanisms (e.g., DataComp-style filtering), but note that filtered captions may still lack visual descriptiveness.
• The reinforced dataset augments each original image-caption pair with additional synthetic and multimodal artifacts (see below).

Key details for each reinforced component

• Synthetic captions: The CoCa model generates multiple synthetic captions per image to boost visual descriptiveness. Ablations on the number of generated captions are reported in the paper. Real captions are more specific but noisier; a combination of both is shown to be crucial for best zero-shot retrieval and classification performance.
• Image augmentations: For each image, multiple augmented versions are created using a parameterized augmentation function, with parameters stored to allow reproducibility. The number and types of augmentations are ablated in the paper.
• Ensemble teacher embeddings: An ensemble of K CLIP models serves as a strong teacher. For each augmented image and each synthetic caption, the authors compute feature embeddings from all K teachers. They also compute teacher embeddings for the ground-truth captions.

Reinforced dataset schema

• Each dataset entry stores: the original image, the original (ground-truth) caption, the augmentation parameters, the synthetic captions, and the teacher feature embeddings for augmented images, synthetic captions, and ground-truth captions.

How the paper uses the data

• Dataset reinforcement is a one-time preprocessing cost, amortized over multiple efficient model training runs and experiments.
• During training, the model consumes the reinforced dataset, using both real and synthetic captions together with teacher embeddings to improve learning. Mixture ratios and the impact of each component are evaluated through ablations (e.g., number of synthetic captions, number and kind of augmentations, teacher choices).

Processing notes

• The augmentation function is parameterized, and the parameters are saved alongside the data so that each augmented image can be reproduced exactly from the original.
• Teacher embeddings are computed offline using the ensemble, ensuring the same reinforced dataset can be reused across experiments without recomputation.

Method

The authors propose a Multi-Modal Reinforced Training framework designed to enhance the training of efficient image-text models, such as MobileCLIP, without incurring the computational overhead of online teacher evaluation. The core strategy involves a one-time dataset reinforcement phase where additional knowledge is generated and stored, followed by a distillation-based training process.

Dataset Reinforcement Pipeline The method begins by augmenting a base image-text dataset (e.g., DataComp) with richer information. For every image, the system generates multiple augmented versions using parametrized augmentation functions. Simultaneously, a strong image captioning model, such as CoCa, is employed to generate multiple synthetic captions for each image, providing more descriptive visual details than the original web-crawled captions.

To transfer knowledge from larger models, the authors utilize an ensemble of strong, pre-trained CLIP models as teachers. The system computes feature embeddings for the augmented images, the synthetic captions, and the ground-truth captions using this teacher ensemble. All this additional information—augmentation parameters, synthetic captions, and teacher embeddings—is stored alongside the original data to create a reinforced dataset (DataComp-DR). This ensures that the heavy computation of generating this knowledge happens only once, allowing for efficient iterative training.

Refer to the framework diagram for the complete pipeline, which illustrates the generation of augmentations and synthetic captions, the computation of teacher embeddings, and the storage of this added knowledge into the reinforced dataset.

Training Process and Loss Function During training, the student model leverages the reinforced dataset to learn from both real and synthetic signals. For each training step, the data loader randomly selects one augmentation and one synthetic caption for a given sample. This allows the construction of two distinct training signals: pairs of (augmented image, real caption) and pairs of (augmented image, synthetic caption).

The training objective combines a standard contrastive loss with a knowledge distillation loss. The total loss is defined as:

The distillation loss aligns the student's output with the teacher ensemble by minimizing the Kullback-Leibler divergence between the similarity matrices of the teacher embeddings and the student embeddings. Since the teacher embeddings are pre-computed and stored in the dataset, the student can compute this loss via a single forward pass without running the teacher models during training.

Model Architecture Design The framework employs specific architectural choices to ensure efficiency, particularly for the text encoder. The authors introduce a hybrid text encoder called Text-RepMixer, which combines 1-D convolutions for token mixing with self-attention layers.

As depicted in the architecture section of the framework diagram, the design decouples the train-time and inference-time structures. During training, the model utilizes standard blocks involving 1x1 depthwise convolutions and BatchNorm layers. For inference, the architecture is reparameterized; the skip connections and BatchNorm layers are fused into the main convolution path. This reparameterization significantly reduces latency while maintaining the representational capacity provided by the self-attention components. The image encoder follows a similar efficiency-focused design, based on FastViT but with a reduced MLP expansion ratio to improve parameter efficiency.

Experiment

The evaluation uses DataComp benchmarks across zero-shot classification, retrieval, and the ARO compositional tasks, with ablations on DataComp-12M and large-scale training on DataComp-1B. Key findings show that strong image augmentations, synthetic captions, and an ensemble teacher significantly boost performance, while a hybrid text encoder offers a favorable speed-accuracy tradeoff. Reinforced training with DataCompDR yields over 100x data efficiency and consistently outperforms prior methods, including larger and longer-trained models, while also improving compositional understanding on ARO.

The table compares text encoder latency and zero-shot ImageNet accuracy between a base transformer and the proposed MCt hybrid text encoder, showing that MCt achieves a substantial latency reduction while slightly improving accuracy. The image encoder comparison with FastViT is also mentioned but not quantified in the provided table data. MCt text encoder lowers latency from 3.3 ms to 1.6 ms while improving zero-shot accuracy from 53.4% to 53.6%. The hybrid text encoder design, combining convolutions and self-attention, proves more efficient than the purely convolutional or base transformer alternatives.

The ablation study isolates the impact of synthetic captions, strong augmentations, and an ensemble teacher on a distilled CLIP model. Adding synthetic captions yields large gains, especially on retrieval, while strong augmentations and an ensemble teacher further boost zero-shot classification. The loss weighting lambda presents a tradeoff between classification and retrieval performance. Synthetic captions improve ImageNet accuracy by 7.4 points and Flickr30k by 27.5 points when used in standard CLIP training. Strong image augmentations add 4.8 points on ImageNet and 4.4 points on Flickr30k when combined with distillation. Using an ensemble teacher contributes a 2.4 point gain on ImageNet over a single teacher. A lambda of 1.0 yields the best ImageNet accuracy, while 0.7 gives the highest Flickr30k retrieval score.

Reducing the number of self-attention layers in the model lowers both parameter count and inference latency, but also decreases ImageNet validation accuracy. The full 6-layer configuration achieves the highest accuracy at 60.9%, while the 0-layer variant drops to 57.9% despite being much smaller and faster. Training on DataCompDR introduces no time overhead relative to the standard setup. Removing self-attention layers progressively reduces model size from 44.5M to 38.3M parameters and latency from 1.9ms to 1.2ms. Accuracy degrades smoothly from 60.9% to 57.9% as self-attention layers are removed, indicating a trade-off between efficiency and performance. The efficiency gains come from fewer self-attention layers, but the accuracy penalty remains modest, with the 0-layer variant still above 57%.

The table compares zero-shot ImageNet accuracy and latency across CLIP variants under similar training budgets. MobileCLIP-B with a hybrid text encoder achieves the highest accuracy (65.3%) when trained on DataCompDR-12M, while maintaining lower text-encoder latency than standard CLIP-B/16. Using the same architecture but less curated data (DataComp-12M) yields 50.1%, showing the importance of data filtering. MobileCLIP-B with the hybrid text encoder reaches 65.3% zero-shot accuracy on DataCompDR-12M, outperforming all compared models. Standard CLIP-B/16 trained on CC-12M or YFCC-15M achieves only about 36-38% accuracy, far below MobileCLIP-B. MobileCLIP-B's text-encoder latency (3.3 ms) is lower than CLIP-B/16's (3.3 ms) while image latency is also reduced (10.4 vs 11.5 ms).

The MobileCLIP family consistently achieves the best average performance across different latency groups, outperforming prior large-scale trained models like TinyCLIP and ViT-based baselines while being smaller and faster. The gains are especially notable in retrieval tasks and on the 38-dataset average, with MobileCLIP models trained on DataCompDR-1B showing strong efficiency and accuracy trade-offs. MobileCLIP-S0 outperforms TinyCLIP variants while having lower latency and fewer parameters. MobileCLIP-S2 surpasses a ViT-B/32-256 model trained with 2.6x longer schedule in average performance and retrieval, while being smaller and faster. MobileCLIP-B achieves higher average performance and better retrieval than SigLIP-B/16 despite being 26.3% smaller and trained on less data. All MobileCLIP models are trained from scratch, unlike TinyCLIP which uses advanced weight initialization from OpenCLIP models.

Across multiple experiments, the proposed MobileCLIP architecture with a hybrid text encoder consistently reduces latency while maintaining or improving accuracy compared to transformer and convolutional baselines. Ablation studies show that synthetic captions, strong augmentations, and ensemble distillation each contribute meaningful gains, with a trade-off between classification and retrieval depending on loss weighting. Removing self-attention layers yields efficiency improvements at a modest accuracy cost, and comparisons under similar training budgets highlight the importance of data curation, as MobileCLIP trained on filtered data outperforms prior models while being smaller and faster. Overall, MobileCLIP variants demonstrate strong efficiency-accuracy trade-offs across different latency groups, particularly in retrieval tasks.