One-sentence Summary

Researchers from Tongji University, Microsoft Corporation, and Macquarie University propose LLM2CLIP, an efficient fine-tuning framework that embeds a large language model into pretrained CLIP through a lightweight adaptor trained on only a few million image-caption pairs, achieving consistent gains over state-of-the-art variants such as EVA02 and SigLIP-2 across linear-probe classification, zero-shot image-text retrieval, segmentation, and detection without large-scale retraining.

Key Contributions

• The paper introduces LLM2CLIP, a lightweight fine-tuning framework that strengthens pretrained CLIP for long and complex captions by turning a large language model into a caption embedding module and connecting it to a pretrained CLIP visual encoder through a small trainable adaptor.
• The integration design freezes the large language model, uses its sentence embeddings as fixed features, and discards the original CLIP text encoder, with ablations showing that direct replacement is more effective than combining or concatenating CLIP and LLM text features.
• Training on only a few million image caption pairs yields substantial improvements over strong CLIP baselines, including SigLIP-2 gains of +1.0/+1.9 for short caption retrieval, +14.8/+15.8 for long caption retrieval, and +11.9/+15.2 for multilingual tasks, along with gains in zero-shot classification, segmentation, detection, and multimodal benchmarks.

Introduction

CLIP is a lightweight cross-modal foundation model that maps images and text into a shared space, supporting image-text retrieval, zero-shot classification, detection, segmentation, and generative pipelines. Its dual-tower design is efficient and scalable, but its text encoder can struggle with long or complex captions and lacks the broad textual understanding of modern LLMs. Prior attempts to integrate LLMs with CLIP often use weaker language encoders, require costly joint training, or overlook that raw LLM embeddings are not discriminative enough for contrastive learning. The authors leverage LLM2CLIP, which first fine-tunes an LLM into a caption-aware embedding model and then replaces CLIP’s text encoder with a frozen LLM plus a small adapter, improving retrieval and downstream multimodal tasks at nearly standard CLIP fine-tuning cost.

Dataset

• The authors use two main data sources for contrastive training: image-derived caption corpora and pure-text pairs.

• Caption data:

  • Source: caption corpora extracted from image-text datasets.
  • Main subset: Dreamlip captions.
  • Key detail: images may have multiple associated captions.
  • Use: supports supervised textual contrastive learning and helps align the LLM embedding space with downstream multimodal tasks.

• Pure-text data:

  • Source: Echo Embeddings.
  • Size: 1.5M pure-text pairs.
  • Use: added to preserve general language understanding during contrastive training.

• Data usage:

  • The caption data and pure-text pairs are combined into a mixed contrastive training setup.
  • This mixture is used to fine-tune the LLM, whose embeddings are later leveraged for CLIP fine-tuning.

Method

The proposed method can be understood as a two stage adaptation pipeline. In the first stage, a large language model is converted into a sentence embedding model through caption based contrastive fine tuning. In the second stage, this fine tuned language model is inserted into a pretrained CLIP model as the text pathway, and CLIP is then post fine tuned using image and text contrastive learning.

The first stage addresses a mismatch between generative language models and embedding models. A standard language model outputs discrete tokens through its output layer, while CLIP requires dense sentence level representations. The authors therefore map the output features of the language model into a whole sentence representation. They compare an end of sequence token representation and average pooling across all output tokens. Average pooling is used as the default because it empirically gives stronger feature quality. The end of sequence token can attend to the full input sentence, but pooling provides a more stable aggregation for embedding extraction.

To improve textual comprehension, the authors modify the attention behavior of the language model. Since the downstream goal is encoding rather than generation, the causal attention mask is removed. This allows the model to model bidirectional relationships among tokens in the input caption. The authors also apply parameter efficient fine tuning with low rank adaptation. This keeps the main language model weights mostly frozen while enabling the output space to adapt to embedding style objectives.

An adaptor is also examined to increase feature separability after the language model. Three variants are considered. The first is a transformer style adaptor with several latent cross attention layers. The second is a linear adaptor formed by multiple linear layers after the sentence representation is extracted. The third uses no adaptor. The adaptor is treated as a compact module that can further reshape the language model embeddings before they are used for contrastive learning.

For training in the first stage, the authors initially consider masked next token prediction. This objective masks selected tokens and predicts them from the position immediately before the masked token, preserving the next token prediction convention of a causal language model. Although such an objective can improve pure text embeddings, the authors find that it underperforms relative to direct contrastive fine tuning in the multimodal setting. Therefore, the core training signal becomes textual contrastive learning.

The textual contrastive learning setup follows the SimCSE style idea of separating positive and negative caption pairs in the embedding space. The authors use two possible forms. The unsupervised form relies on dropout augmentation to create different views of the same sentence. The supervised form uses semantically related caption pairs from annotated data. The supervised variant is used as the default because it provides stronger supervision. Positive pairs are constructed from two distinct captions of the same image. These pairs are formatted with a system prompt that asks the model to retrieve a similar relevant caption from a given caption.

Given a batch of caption representations, the supervised contrastive objective pulls the representation of an anchor caption closer to its positive pair and pushes it away from other captions in the batch. Let $\mathbf{z}_i$zi​ denote the embedding of an anchor caption, $\mathbf{z}_i^{+}$zi+​ its positive caption, and $\tau$τ the temperature scaling factor. A typical contrastive loss over an in batch set of negatives can be written as

The first stage uses caption corpora derived from image and text datasets. Caption collections that provide multiple captions per image are particularly useful because they naturally provide supervised positive pairs. To avoid overfitting the language model to short caption style text, the authors also mix pure text contrastive pairs into training. This helps preserve broader language understanding while still adapting the embedding space toward caption retrieval semantics.

The second stage begins from a pretrained CLIP model whose visual and language components already encode a strong image language mapping. CLIP typically consists of a vision encoder and a comparatively small text encoder. The authors treat this pretrained mapping as a strong foundation and introduce the caption contrastive fine tuned language model to enrich the language side. The goal is not to train CLIP from scratch, but to perform a targeted post fine tuning step that connects the large language model to the visual embedding space.

Efficient adaptation is central to this stage because CLIP training normally requires large datasets and large batch sizes. The authors use parameter efficient fine tuning. For the visual encoder, they allow full gradients by default so that visual representations can learn from the richer language side signals. For the language model, they compare two options. The first keeps the language model trainable with low rank adaptation. The second freezes the language model completely and attaches an adaptor after its output layer. The frozen language model with adaptor is used as the default configuration, specifically an adaptor composed of four linear blocks.

The frozen language model adaptor design provides an important system benefit. Because the language model is frozen, its text features can be precomputed offline and stored during training. This avoids repeatedly loading the large language model into GPU memory across multiple training epochs. The training loop therefore only needs to load CLIP, the adaptor, and the cached sentence features, which reduces GPU memory usage, lowers inference overhead, and accelerates experimentation.

After adding the language model, CLIP has two potential text pathways: the original CLIP text encoder and the new language model encoder. The authors explore several ways to combine them. One option removes the original CLIP text encoder and directly pairs the language model with the CLIP vision encoder for cross modal contrastive learning. Another keeps both text encoders and applies separate contrastive losses to each. A third keeps both and adds an additional contrastive task between the language model and the original text encoder. A fourth concatenates their output features and performs contrastive learning with the vision encoder. The adopted design is the first option: the original CLIP text encoder is removed, and the fine tuned language model embedding is used as the sole text encoder.

In this cross modal contrastive setup, the vision encoder produces an image embedding, while the frozen language model and adaptor produce a text embedding. The model then optimizes a symmetric contrastive alignment between image and text representations within an image text batch. If $\mathbf{v}_i$vi​ is the embedding of image $i$i, $\mathbf{t}_i^{+}$ti+​ is the embedding of its corresponding caption, and $\tau$τ is the temperature, the contrastive objective encourages matched image and text pairs to occupy nearby regions in the shared space.

The second stage training data focuses on dense and detailed image captions. Because the language model is much stronger than the original CLIP text encoder at processing long textual descriptions, the method can benefit from captions that contain richer visual details. The authors use caption variants that provide multiple dense descriptions for the same image. During training, they examine multiple positive caption examples for a single image and blend real captions with dense captions generated by multimodal captioning systems. The final default setting blends real captions and dense generated captions at a fifty percent ratio, allowing the model to learn from both natural human descriptions and detailed synthetic descriptions.

Experiment

The experiments evaluate a two-stage LLM2CLIP pipeline across retrieval, classification, segmentation, detection, and multimodal model tasks, using CLIP-family encoders, different data scales, and alternative language backbones. Overall, LLM2CLIP improves cross-modal alignment, especially for long-text and multilingual retrieval, and yields stronger visual representations that benefit downstream probing and multimodal reasoning, although zero-shot classification shows a modest trade-off. Ablations further show that supervised SimCSE with LoRA is essential in the first stage, additional training data consistently helps, and the second stage works best with a simple linear adaptor and by replacing rather than reusing the original CLIP text encoder.

The comparison shows that LLM2CLIP post training strengthens several pretrained CLIP style models on zero shot English text image retrieval. The gains appear for both short caption and long caption benchmarks, with larger advantages on long caption datasets. The improvement is observed across multiple visual model variants and even for strongly pretrained SigLIP2 models. LLM2CLIP yields consistent retrieval improvements over base CLIP, EVA02, and SigLIP2 models. Long text retrieval datasets benefit more than short caption datasets such as Flickr and COCO. The method remains effective across different model sizes and visual backbones.

The multilingual retrieval results show that adding LLM2CLIP consistently improves bidirectional image-text retrieval on Flickr-CN, COCO-CN, and XM3600. The gain is especially pronounced for EVA-L-224, which performs poorly on the Chinese short-caption datasets before post-training but becomes competitive after LLM2CLIP. SigLIP2 is already strong, yet LLM2CLIP still raises both image-to-text and text-to-image scores. LLM2CLIP produces broad gains across Chinese Flickr and COCO retrieval and XM3600. EVA-L-224 changes from weak Chinese retrieval to strong bidirectional retrieval after LLM2CLIP. LLM2CLIP improves SigLIP2 across all reported SigLIP2 comparisons despite its already high baseline.

The table compares CLIP baselines with their LLM2CLIP post-trained variants on ImageNet zero-shot classification and linear probe evaluation. Across both model sizes, LLM2CLIP slightly lowers zero-shot accuracy while slightly improving linear probe performance. The larger 336-resolution CLIP variant remains stronger than the standard CLIP variant in all reported settings. LLM2CLIP provides small gains in linear probe performance over the corresponding CLIP baselines. Zero-shot classification accuracy does not improve with LLM2CLIP in this table. CLIP L/14-336 consistently outperforms CLIP L/14 in both zero-shot and linear probe settings.

The reported results show that adding LLM2CLIP to the EVA02 visual model improves performance on zero-shot segmentation and open-vocabulary detection tasks. The gains are consistent across the segmentation datasets and are more pronounced than the smaller but still positive changes in detection accuracy and COCO AP metrics. Zero-shot segmentation mIOU increases across COCO-S, ADE, VOC, and City, with especially strong relative gains on VOC and City. Open-vocabulary COCO detection also improves, including modest increases for novel, base, and all categories as well as detection AP.

The table compares Llava 1.5 results when the original CLIP ViT-L/14 visual encoder is replaced by the fine-tuned LLM2CLIP version. The +LLM2CLIP row generally outperforms the reproduced Llava baseline on several benchmark columns, especially Seed V2, GQA, Vz, TV, MB, MC, and LB. A few columns, such as SQA and MV, are lower than the reproduced baseline, indicating that the improvement is broad but not uniform. Replacing CLIP ViT-L/14 with the fine-tuned LLM2CLIP variant raises VQA, POPE, MM, Seed V2, GQA, and Vz scores relative to the reproduced Llava run. The +LLM2CLIP model also improves TV, R, A, P, MME, MB, MC, and LB relative to the reproduced Llava baseline. SQA and MV remain lower for +LLM2CLIP than for the reproduced Llava baseline, so the replacement does not improve every benchmark column.

The experiments evaluate LLM2CLIP as a post-training method for CLIP-style visual encoders, testing its effect on zero-shot text-image retrieval in English and Chinese, ImageNet classification, open-vocabulary segmentation and detection, and a Large Language-and-Vision Assistant with a swapped visual encoder. The results show that LLM2CLIP generally strengthens retrieval performance across model sizes and visual backbones, with especially strong benefits for long-caption and multilingual settings and notable recovery for weaker Chinese retrieval baselines. Although ImageNet zero-shot classification accuracy decreases slightly, linear probe performance improves, and the method also supports gains in open-world visual understanding tasks and most multimodal assistant benchmarks, indicating broad but not uniformly positive effectiveness.