One-sentence Summary

Tencent's WeChat Vision team introduces WeMM-Embedding, a family of universal multimodal embedding models with 2B, 4B, and 9B variants that support text, images, videos, visual documents, and interleaved inputs with flexible output dimensions, are trained via two-stage alignment and cross-scale knowledge transfer, achieve state-of-the-art results including an 80.6 overall score on MMEB-v2, and are deployed across WeChat recommendation and search applications.

Key Contributions

• This work introduces WeMM-Embedding, a family of universal multimodal embedding models at 2B, 4B, and 9B scales that supports text, images, videos, visual documents, and interleaved multimodal inputs with flexible output dimensions.
• A two-stage training strategy uses a large-scale multimodal alignment stage followed by refinement with curated data, fine-grained relevance supervision, and cross-scale knowledge transfer.
• The 2B variant surpasses the previous leading 8B open-source baseline on MMEB-v2, and the 9B variant achieves a state-of-the-art overall score of 80.6. The models also show gains on a 26-task in-house benchmark and consistent improvements across 14 online A/B tests, and they are deployed at scale across WeChat recommendation and search applications; model weights and code are released.

Introduction

Multimodal embedding models are central to retrieval, recommendation, classification, and agentic systems because they map text, images, and videos into a shared representation space. CLIP-style dual encoders and later encoder-based any-to-any models are effective, but they remain limited when inputs mix modalities or when tasks demand more general joint representations. Multimodal large language models offer a more flexible foundation, yet turning their hidden states into strong universal embeddings still requires careful alignment and relevance training. The authors introduce WeMM-Embedding, a 2B to 9B model family trained in two stages: first on massive heterogeneous multimodal pairs for broad alignment, then on a curated corpus with harder negatives, richer relevance supervision, and cross-scale knowledge transfer for finer-grained matching.

Dataset

Dataset composition and processing

• Overall scale and sources: The authors collect and synthesize several hundred million training examples spanning text, images, videos, and interleaved multimodal inputs. Sources include public datasets, web-scale weakly supervised sources, task-oriented synthetic data, and in-house collections.
• Unified schema: Each example is represented as $z_i = (I_i, q_i, c_i, \mathcal{N}_i, y_i)$zi​=(Ii​,qi​,ci​,Ni​,yi​), where $q_i$qi​ is the source instance, $c_i$ci​ is the paired target, $I_i$Ii​ is an optional task instruction, $\mathcal{N}_i$Ni​ is an optional set of hard negatives, and $y_i$yi​ is an optional graded relevance score. Both $q_i$qi​ and $c_i$ci​ can contain text, images, videos, or interleaved combinations.
• Main data types and supervision:
  • Weakly supervised pairs: Large-scale image-text and video-text pairs from public and web-scale sources, with naturally occurring correspondence rather than explicit descriptions.
  • Caption pairs: Images and videos paired with explicit captions describing entities, attributes, relations, spatial context, actions, and events.
  • Retrieval pairs: Queries paired with relevant candidates across text, images, videos, and interleaved inputs, including composed queries, reasoning, long contexts, spatial grounding, temporal localization, and agent-related retrieval.
  • Classification pairs: Classification datasets reformulated as source-label pairs, where the target is a class name or a natural-language category description.
  • Multimodal QA pairs: Textual or multimodal questions paired with answer targets for visual perception, relational and spatial understanding, OCR, knowledge-intensive understanding, reasoning, document and chart comprehension, and event understanding.
  • Graded relevance pairs: Source-target pairs with manually assigned discrete relevance levels, supporting ranking-oriented training.
• Large-scale processing: All examples are converted into the shared pair-based format and organized into task-specific batches. Standard pairwise data use other targets in the batch as in-batch negatives. For shared target spaces, such as classification, false negatives are removed through semantic target masking.
• Curated subset: A curated collection approximately one tenth the size of the large-scale data is constructed to improve semantic balance, data quality, and supervision.
  • Semantic-ID-guided resampling: Each pair is embedded through an intermediate WeMM-Embedding checkpoint, then quantized with a three-level residual k-means model to produce a three-element Semantic ID. Examples from dense semantic codes are downsampled, while examples from sparse codes are retained at higher rates.
  • Quality filtering: A multimodal large language model checks whether each source-target pair matches the intended relation and filters mismatched examples. It also refines text fields where needed, such as correcting factual inconsistencies in weakly supervised image-text and video-text pairs while preserving alt-text style and detail.
  • Hard-negative construction: For text targets, multimodal LLMs generate plausible but incorrect candidates. For image and video targets, intermediate WeMM-Embedding checkpoints retrieve semantically similar candidates from task-specific candidate pools. A smaller subset receives reranker-derived relevance scores for finer-grained supervision.
• Use in the model: Both the large-scale and curated collections are used within a unified multi-task training pipeline. Graded relevance values are used to construct relative ordering constraints, and explicit hard negatives improve discrimination among similar candidates. The provided excerpt does not report a separate train/validation split or exact mixture ratios beyond the curated set being about one tenth the size of the large-scale collection.
• Cropping or augmentation details: Image-specific cropping and augmentation details are not specified in this excerpt. The main processing focus is unified pair formatting, semantic resampling, quality filtering, and hard-negative enrichment.

Method

Model Architecture

The authors build WeMM-Embedding as a family of universal multimodal embedding models with 2B, 4B, and 9B parameters, all based on natively multimodal Qwen3.5 backbones. The architecture is designed to encode arbitrary combinations of text, images, and videos into unified dense representations. An input instance may contain an optional task-specific instruction together with an ordered sequence of multimodal segments:

where $I_{\mathrm{inst}}$Iinst​ denotes the optional instruction and each $x_i$xi​ may be text, an image, or a video. Textual segments are converted into token embeddings through the native tokenizer and embedding layer, while visual segments are processed through the native visual pipeline. The token sequence is arranged in the original segment order, and a dedicated <embedding> token is appended:

The full multimodal sequence is then processed by the LLM backbone $G_\theta$Gθ​:

The final-layer hidden state of the <embedding> token is used for last-token pooling and is L2-normalized to produce the output representation:

The causal attention formulation also allows multiple <embedding> tokens to be inserted at different positions. For example, for a video followed by its automatic speech recognition transcript, one token can be placed after the video tokens and another at the end of the sequence. This enables video-only and joint video-text representations to be extracted in a single forward pass.

WeMM-Embedding further supports flexible embedding dimensions through Matryoshka Representation Learning. Given the final hidden state $\mathbf{h}_{\mathrm{emb}} \in \mathbb{R}^D$hemb​∈RD, an embedding of dimension $d \leq D$d≤D is obtained by retaining the first $d$d dimensions and applying L2 normalization:

This allows embeddings at multiple supported dimensions to be produced from a single forward pass through prefix truncation and re-normalization.

Curated Data Construction

In addition to large-scale collection, the authors construct a curated dataset focused on semantic balance, data quality, and more informative supervision. Construction includes Semantic-ID-guided resampling, quality control, and selective hard-negative enrichment.

For Semantic-ID-guided resampling, each source-target pair is mapped to a discrete Semantic ID. The side with the longer serialized token sequence is encoded using an intermediate checkpoint of WeMM-Embedding. A three-level residual k-means quantizer is then fitted to the resulting representations, producing a three-element Semantic ID. Examples assigned to densely populated codes are sampled at lower rates, while examples mapped to less populated codes are retained at higher rates. This reduces repeated exposure to frequent semantic patterns.

The resampled examples are further refined using a multimodal large language model. The model assesses whether each source-target pair reflects the intended matching relation and filters out mismatched examples. It also corrects noisy or factually inaccurate textual descriptions while preserving the original style and level of detail.

A subset of refined examples is enriched with explicit hard negatives. For text targets, multimodal large language models generate plausible but incorrect candidates based on the source and positive target. For image and video targets, intermediate WeMM-Embedding checkpoints retrieve semantically similar candidates from task-specific candidate pools. A smaller subset of mined candidates is further scored by reranking models to provide finer-grained supervision among difficult candidates.

Stage 1: Large-Scale Multimodal Alignment

The first training stage establishes a general multimodal embedding space using large-scale multi-task alignment. Each batch is constructed from a consistent data source, while batches from different tasks are interleaved throughout training. Standard paired examples are optimized with contrastive learning, and examples with native graded relevance annotations use a score-gap-weighted ranking objective.

For standard paired data, the authors optimize source-target alignment with an InfoNCE objective. Each source uses its paired target as the positive and targets associated with other examples as in-batch negatives. When explicit hard negatives are available, they are added to the negative pool. Let $\mathcal{C}_j = \{c_j^+\} \cup \mathcal{N}_j$Cj​={cj+​}∪Nj​ denote the candidates associated with source $q_j$qj​. The contrastive objective is:

where $s(\cdot,\cdot)$s(⋅,⋅) denotes cosine similarity between normalized embeddings and $\tau$τ is a learnable temperature. To handle collisions from near-duplicate sources or targets, duplicate-aware masking is applied:

Source-side masking excludes candidates associated with a near-duplicate source, while target-side masking excludes candidates that closely match the current positive target.

For pairs annotated with discrete relevance levels, the authors use a score-gap-weighted CoSENT-style ranking objective. For two examples $i$i and $j$j, the relevance-gap weight is

and the ranking loss for example $i$i is

The batch objective is the average over all examples. This encourages higher-relevance pairs to receive higher similarities and gives larger weight to comparisons with larger label gaps.

Matryoshka Representation Learning is applied to both contrastive and graded-relevance training. For each batch, the corresponding objective is evaluated independently at every supported embedding dimension:

Stage 2: Curated Fine-Tuning and Distillation

The second stage continues training on the curated dataset. The contrastive and graded-relevance objectives remain in use, while additional supervision is provided by reranking models and a larger embedding teacher.

Dedicated multimodal rerankers provide query-specific ordering signals over candidate sets containing a positive target and mined hard negatives. Reranker scores replace manually assigned relevance levels, and comparisons are constructed only among candidates associated with the same query. For source $q_b$qb​ with candidate set $\mathcal{C}_b$Cb​, the reranker scores induce the ordered pair set

The reranker objective is

where $\omega_{b,ij}=\max(|\hat{y}_{b,i}-\hat{y}_{b,j}|,\epsilon)$ωb,ij​=max(∣y^​b,i​−y^​b,j​∣,ϵ). Reranker supervision is restricted to settings where it yields reliable improvements.

Embedding distillation transfers online soft targets from a larger teacher model using batch-wise source-target similarity distributions. For a batch with $B$B sources and a target pool of size $K$K, the teacher and student similarity matrices are defined as

Reverse similarity matrices between positive targets and sources are also constructed:

Row-wise relation distributions are obtained by softmax, and the bidirectional distillation loss is

This aligns teacher and student similarity distributions in both source-to-target and target-to-source directions. Unlike one-hot contrastive supervision, the teacher distributions preserve relative similarity differences among candidates.

For each Stage 2 batch, the task objective is selected according to available supervision:

The overall Stage 2 objective is then

where $\lambda_{\mathrm{Emb}}$λEmb​ controls the distillation weight. For the 2B and 4B variants, the frozen 9B WeMM-Embedding model serves as the embedding teacher. For the 9B variant, multiple specialized Stage 2 variants are trained with complementary data mixtures and configurations, then combined through model merging.

Experiment

WeMM-Embedding is evaluated on a comprehensive benchmark suite spanning the MMEB series for general multimodal and agent-centric tasks, public cross-modal retrieval datasets, and real-world WeChat applications. The model establishes leading performance across image, video, visual-document, text, and agent retrieval, frequently surpassing larger open-source and proprietary baselines. Ablation and cumulative studies reveal that task-consistent batching, curated data, reranker supervision, and embedding-teacher distillation are key to refining universal multimodal representations, while Matryoshka learning preserves high performance at reduced dimensions for efficiency. Real-world deployment in large-scale recommendation and search systems consistently improves content matching and user engagement, confirming practical value beyond benchmarks.

Among the 2B-parameter multimodal embedding models evaluated, RzenEmbed achieves the highest overall score, driven by exceptional performance in image classification and visual grounding. VLM2Vec-V2 and Ops-MM-embedding-v1 both surpass 59 points overall, with VLM2Vec-V2 markedly improving over its predecessor, especially on visual documents. The baseline results reveal large gaps in retrieval and visual grounding across models, highlighting the challenge of these tasks for compact multimodal representations. RzenEmbed leads all 2B models with an overall average of 67.2, outperforming the next best, Ops-MM-embedding-v1, by 2.6 points. Image classification sees the widest spread: RzenEmbed reaches 90.3, while GME only achieves 55.5. VLM2Vec-V2 improves upon the original VLM2Vec by 11.5 points overall, with its visual document score nearly doubling from 44.0 to 69.2. Retrieval remains a weak spot for most 2B models; GME scores just 29.9 on image retrieval, and even the best performer, Ops-MM-embedding-v1, reaches only 47.6.

WeMM-Embedding 2B attains the highest overall score on MMEB-v3 (56.0), surpassing all listed baselines by a clear margin while scoring zero on audio tasks due to lack of audio input support. Its advantage is driven by strong text and agent-centric performance, with particularly large gains in instruction following, long-context retrieval, and tool- and memory-focused agent subtasks. Larger variants further extend this lead. WeMM-Embedding 2B achieves a V3-All score of 56.0, outperforming the next best model (Qwen3-VL-Embedding, 50.9) by over 5 points despite not supporting audio. On the 53 text tasks, WeMM-Embedding reaches 45.3, exceeding all baselines, with instruction following (49.3) and long-context retrieval (61.9) showing especially large margins. Agent task performance is highest at 45.1, with tool retrieval (46.2) and memory retrieval (47.2) showing substantial improvements over Qwen3-VL-Embedding's 42.6 and 28.4. Audio tasks receive zero for WeMM-Embedding and several baselines that lack audio capability, while Omni-Embed-Nemotron and E5-Omni achieve nonzero audio scores. Scaling to 4B and 9B lifts V3-All to 58.2 and 59.5 respectively, widening the gap over existing models according to the paper's analysis.

Across the reported cross-modal retrieval benchmarks, WeMM-Embedding variants are competitive with proprietary commercial embedding models. The WeMM-Embedding 9B model achieves the best text-to-image recall on MSCOCO and DOCCI, and WeMM-Embedding 4B achieves the best image-to-text recall on MSCOCO among the listed models. On mean text-to-image recall, WeMM-Embedding 9B is nearly on par with the top commercial model, while the 2B model already surpasses the larger Qwen3-VL-Embedding. WeMM-Embedding 9B records the highest text-to-image retrieval scores on MSCOCO and DOCCI, outperforming proprietary models on those benchmarks. WeMM-Embedding 4B leads image-to-text retrieval on MSCOCO, and WeMM-Embedding 2B beats the larger Qwen3-VL-Embedding 8B on mean text-to-image recall.

WeMM-Embedding outperforms the Qwen3-VL-Embedding baseline on the in-house benchmark, achieving a higher average score and higher scores in all five task categories. The advantage is especially clear for cross-domain content matching and classification, while video relevance shows a comparatively smaller gap. WeMM-Embedding leads across classification, search, cross-domain content matching, article relevance, and video relevance. Cross-domain content matching shows the largest category-level improvement over the baseline, and video relevance shows the smallest.

A small-scale Stage-1 ablation with a 2B model shows that each evaluated design choice contributes to MMEB-v2 performance. Task-consistent batching has the largest effect, with its removal lowering the overall average by 3.4 points, while removing task-specific instructions and duplicate-aware masking produce smaller but consistent declines. The full configuration achieves the highest average performance across image, video, and visual-document tasks. Removing task-consistent batching causes the largest overall drop, with image tasks degrading the most among domains. Task-specific instructions matter most for visual-document tasks, where their removal leads to a larger decline than in image or video tasks. Duplicate-aware masking provides a modest benefit, and its removal slightly reduces overall and image performance.

The evaluation spans multiple multimodal embedding benchmarks, including MMEB-v3, cross-modal retrieval, and an in-house test suite, to assess compact models and validate design choices. Among 2B-parameter models, RzenEmbed leads overall with strong image classification and grounding, while retrieval remains a common weakness. WeMM-Embedding variants achieve top scores on text and agent-centric tasks and scale effectively, with the 9B model rivaling proprietary systems on text-to-image retrieval. Ablations confirm that task-consistent batching, task-specific instructions, and duplicate-aware masking all contribute to performance, with batching having the largest impact.