One-sentence Summary

KAIST AI proposes I0T, which identifies modality-specific encoder characteristics as the source of the modality gap in CLIP and addresses it through two variants, a post-hoc standardization ($\mathrm{I0T_{post}}$I0Tpost​) and a trainable method with normalization layers, both reducing the gap to near zero while preserving original embeddings and offering an explainable alternative to CLIPScore.

Key Contributions

• The analysis categorizes the CLIP modality gap using cosine distance, length scale, and mean cosine distance, and shows that persistent peak activations in normalized image and text encoder embeddings theoretically limit cosine similarity and induce the gap.
• The paper introduces I0T_post, a post-hoc embedding standardization method that reduces the modality gap to approximately zero without retraining, preserves the semantic embedding representations of trained models, and can serve as an explainable alternative to CLIPScore.
• The paper introduces I0T_async, a trainable method that adds separate normalization layers for each modality, and experiments show that I0T_post and I0T_async reduce the modality gap and improve downstream performance compared with seven CLIP-based models and BLIP, with no additional training parameters for I0T_post and 10M extra training parameters for I0T_async.

Introduction

Vision language models such as CLIP are widely used for multimodal retrieval, captioning evaluation, and other zero shot tasks, but their image and text embeddings often occupy separate regions of the latent space. This modality gap makes cross modal similarity scores less reliable, especially in metrics like CLIPScore, where correct image text pairs can receive lower scores than irrelevant same modality pairs. Earlier methods mainly try to move embeddings closer or modify training objectives, but they do not fully identify the embedding patterns that cause the gap. The authors show that each encoder develops modality specific activation peaks, then propose the I0T framework to reduce the gap while preserving semantic performance through post hoc embedding standardization and an asynchronous training variant, also introducing I0TScore for cross modal evaluation.

Method

The authors propose a two stage framework, I0T, for vision language models such as CLIP. The overall goal is to reduce the separation between image and text embeddings while preserving the semantic information learned by the encoders. The framework is organized into two stages. The first stage is a plug and play semantic fine tuning module that can be skipped if the user only wants to address the modality gap. The second stage operates asynchronously after the first stage and focuses on removing modality specific characteristics from the normalized image and text embeddings.

In the first stage, the authors improve the semantic representations of CLIP by combining several fine tuning strategies. They follow the implementation style of Long CLIP, but use only long captions for alignment rather than a mixed caption setup. This reduces training cost while maintaining strong downstream performance. The training objective combines standard contrastive learning with cyclic alignment losses, which encourage image and text representations to remain mutually consistent. The combined objective is written as

$$\mathcal{L}_{\mathrm{CyCLIP}} = \mathcal{L}_{\mathrm{CLIP}} + 0.25 \mathcal{L}_{\mathrm{I cyclic}} + 0.25 \mathcal{L}_{\mathrm{C cyclic}}.$$LCyCLIP​=LCLIP​+0.25LIcyclic​+0.25LCcyclic​.

The authors fine tune the CLIP encoders for three epochs using the AdamW optimizer with a learning rate of $1e{-}6$1e−6 and a weight decay of $1e{-}2$1e−2. The batch size is set to 128, and the contrastive temperature uses a log scale of 4.6052. This stage is designed to strengthen semantic alignment without directly attempting to eliminate the modality gap.

The second stage is applied after the semantic encoder is frozen. The authors first examine a post hoc way to reduce the modality gap. They observe that simple clipping of large activations is insufficient because the remaining activations can still contain shared directions that are characteristic of either the image encoder or the text encoder. To remove these shared modality specific patterns, the authors standardize the normalized embedding activations across all hidden dimensions. Let $\mathbf{x}_i$xi​ and $\mathbf{y}_i$yi​ be normalized image and text embedding activations, and let $\bar{\mathbf{x}}$xˉ and $\bar{\mathbf{y}}$yˉ​ be the mean vectors estimated for the image and text modalities. The standardized activations are

$$\mathbf{x}_{i}^{\prime} = \mathrm{Normalize}(\mathbf{x}_{i} - \bar{\mathbf{x}}), \qquad \mathbf{y}_{i}^{\prime} = \mathrm{Normalize}(\mathbf{y}_{i} - \bar{\mathbf{y}}).$$xi′​=Normalize(xi​−xˉ),yi′​=Normalize(yi​−yˉ​).

This operation removes the mean direction associated with each modality and then renormalizes the result. The comparison of the original activations, clipped activations, the proposed standardized activations, and a prior shifting method is shown below.

After standardization, the mean activations become close to zero across all hidden dimensions, and the variance patterns become more modality invariant. The authors argue that this is stronger than methods that only shift embedding clusters, because their approach directly removes the underlying activation patterns that contribute to the gap.

Although the post hoc method is effective, it requires estimating modality specific mean vectors from available data and therefore does not naturally support zero shot inference for a single sample. To address this limitation, the authors propose a learnable asynchronous method. They add two independent batch normalization layers, one for the image encoder and one for the text encoder. These layers are placed after the frozen encoders, so the original semantic parameters remain fixed while the new normalization parameters learn the modality specific means and variances.

During training, the model updates running means and variances for the normalized image and text activations. If $\bar{\mathbf{x}}_B$xˉB​ and $\bar{\mathbf{y}}_B$yˉ​B​ are the batch mean vectors, the running means are updated as

$$\bar{\mathbf{x}}_{t+1} = \alpha \bar{\mathbf{x}}_{B} + (1-\alpha)\bar{\mathbf{x}}_{t}, \qquad \bar{\mathbf{y}}_{t+1} = \alpha \bar{\mathbf{y}}_{B} + (1-\alpha)\bar{\mathbf{y}}_{t}.$$xˉt+1​=αxˉB​+(1−α)xˉt​,yˉ​t+1​=αyˉ​B​+(1−α)yˉ​t​.

The authors use an averaging factor of $\alpha = 0.1$α=0.1. The final running statistics after training, denoted by $\bar{\mathbf{x}}_{\mathrm{train}}$xˉtrain​, $\bar{\mathbf{y}}_{\mathrm{train}}$yˉ​train​, $\sigma_{\mathbf{x}_{\mathrm{train}}}$σxtrain​​, and $\sigma_{\mathbf{y}_{\mathrm{train}}}$σytrain​​, serve as learned modality specific characteristics. At inference, the transformed embeddings are obtained from the learned batch normalization statistics and affine parameters as

$$\hat{\mathbf{x}}_{i} = \mathrm{Normalize} \left( W_{\mathrm{img}} \left( \frac{\mathbf{x}_{i} - \bar{\mathbf{x}}_{\mathrm{train}}} {\sqrt{\sigma_{\mathbf{x}_{\mathrm{train}}} + \epsilon}} \right) + \mathbf{b}_{\mathrm{img}} \right),$$x^i​=Normalize(Wimg​(σxtrain​​+ϵ​xi​−xˉtrain​​)+bimg​), $$\hat{\mathbf{y}}_{i} = \mathrm{Normalize} \left( W_{\mathrm{txt}} \left( \frac{\mathbf{y}_{i} - \bar{\mathbf{y}}_{\mathrm{train}}} {\sqrt{\sigma_{\mathbf{y}_{\mathrm{train}}} + \epsilon}} \right) + \mathbf{b}_{\mathrm{txt}} \right).$$y^​i​=Normalize(Wtxt​(σytrain​​+ϵ​yi​−yˉ​train​​)+btxt​).

Here $W_{\mathrm{img}}$Wimg​, $W_{\mathrm{txt}}$Wtxt​, $\mathbf{b}_{\mathrm{img}}$bimg​, and $\mathbf{b}_{\mathrm{txt}}$btxt​ are the affine weights and biases of the image and text normalization layers. The epsilon value is set to $1e{-}5$1e−5 for numerical stability.

The authors train these normalization layers asynchronously after freezing the fine tuned encoders. This separation is central to the design because it prevents the gap reduction module from rewriting the semantic content already encoded by the CLIP encoders. The batch normalization layers therefore learn only the statistical offsets and scales associated with each modality.

To make the learned statistics more robust, the authors introduce a multimodal contrastive augmentation procedure based on unsupervised positive pairs. Dropout with rate 0.1 is applied to every multi-head attention layer in both the image and text encoders. This creates augmented image and text features, and the training loss is computed over all combinations of original and augmented features. Conceptually, the loss can be written as

$$\mathcal{L}_{\mathrm{MCSIE}} = \sum_{ \mathcal{E}^{I}\in\{\mathcal{I},\mathcal{I}_{\mathrm{aug}}\}, \mathcal{E}^{T}\in\{\mathcal{T},\mathcal{T}_{\mathrm{aug}}\} } \mathcal{L}(\mathcal{E}^{I}, \mathcal{E}^{T}).$$LMCSIE​=EI∈{I,Iaug​},ET∈{T,Taug​}∑​L(EI,ET).

This strategy enlarges the set of positive multimodal associations and helps the normalization layers learn modality specific statistics more reliably. The final architecture preserves the semantic encoder parameters while allowing the added normalization branches to absorb and neutralize the activation patterns that produce the image text embedding gap.

Experiment

The experiments first assess whether CLIP-based training and post-hoc alignment can reduce image-text modality gaps while preserving semantic quality across retrieval, classification, and human-ranking correlation tasks. Results show that the proposed asynchronous and post-hoc variants substantially shrink the gap without degrading downstream effectiveness, with the post-hoc method being especially strong in retrieval and competitive on correlation and classification benchmarks. The findings also support using the post-hoc model as an interpretable, reference-free evaluation metric and demonstrate that its alignment benefits saturate with a small sample subset.

Across Long-CLIP variants, adding cyclic losses improves downstream retrieval while the modality gap remains severe. MCSIE slightly reduces the gap but also slightly lowers downstream performance, whereas layer normalization does not meaningfully relieve the gap. Encoder-specific batch normalization produces a moderate gap, and the asynchronous variant performs best on gap reduction and retrieval. Cyclic-loss training can improve retrieval without eliminating the modality gap. Asynchronous batch normalization per encoder offers a stronger balance between lower modality gap and retrieval performance.

The comparison shows that reducing the modality gap can preserve competitive downstream performance, especially on retrieval tasks. The baseline model exhibits a severe modality gap, while the examined variants reduce the gap to low or moderate levels without broadly harming semantic evaluation. The results also suggest that gap reduction does not guarantee the best outcome on every downstream task. The baseline model shows a severe modality gap, while the gap-reduction methods produce much lower separation measures. The methods with reduced modality gap remain competitive on image-to-text and text-to-image retrieval. Classification and correlation performance does not consistently follow gap reduction, so gap size alone does not fully predict downstream quality.

Using more test samples for the post-hoc embedding shift steadily reduces the modality gap, with severity remaining low across all tested sample sizes. Retrieval performance improves rapidly at small sample sizes and then saturates, so large subsets are not necessary to achieve competitive image-text retrieval. The best retrieval results are obtained when the full test sample set is used. A small test subset can already produce a low modality gap severity level. Retrieval gains become modest after the sample count grows beyond small subsets. Using the entire test set gives the strongest overall retrieval performance.

The experiments evaluate gap-reduction strategies for Long-CLIP, including cyclic losses, MCSIE, layer normalization, encoder-specific batch normalization, and asynchronous batch normalization. Cyclic losses and asynchronous encoder-specific normalization improve retrieval while reducing modality gap, whereas MCSIE and layer normalization offer weaker or less favorable trade-offs. Additional analyses show that post-hoc embedding shift with modest test samples already yields low gap severity and near-saturated retrieval, though using the full set gives the best results. Overall, reducing the modality gap supports competitive retrieval but does not uniformly guarantee better performance across all downstream tasks.