One-sentence Summary

Researchers from CASIA, Alibaba Group, and other institutions propose UEmbed, a decoder-only multimodal embedding model that unifies sparse lexical and dense representations in a single causal forward pass by partitioning the vocabulary into $N$N disjoint subsets and using N learnable special tokens, with UEmbed-9B achieving 71.8 (dense) and 71.0 (sparse) on MMEB-v2, setting the state of the art for sparse retrieval and leading dense models trained on public data, while remaining competitive on BEIR and enabling effective, efficient, and agentic applications.

Key Contributions

• The paper introduces UEmbed, a decoder-only model that produces both dense and sparse retrieval representations in a single causal forward pass by partitioning the vocabulary and using learnable special tokens. UEmbed achieves 71.8 dense and 71.0 sparse on MMEB-v2 and remains competitive with SPLADE-v3 on BEIR.
• UEmbed unifies text and multimodal retrieval without auxiliary cross-modal modules, establishing the first sparse retrieval state of the art on MMEB-v2 and demonstrating cross-modal lexical activations that capture semantic meaning.
• The sparse mode improves efficiency, retains compatibility with high-throughput serving stacks and inverted indices, and reduces tool-call costs on BrowseComp-Plus while maintaining recall. Hybrid dense-sparse scoring further boosts retrieval quality.

Introduction

Learned sparse retrieval (LSR) improves over BM25 by using neural models to produce contextualized sparse lexical representations, but existing LSR methods are tied to bidirectional encoders, lack native multimodal support, and are rarely evaluated on practical efficiency. The authors introduce UEmbed, a decoder-only multimodal model that produces both dense and sparse embeddings in a single causal forward pass. By appending learnable special tokens each assigned a disjoint vocabulary subset via k-means clustering, UEmbed circumvents the representational bottleneck of projecting from a single token. This design enables native sparse retrieval from a causal backbone, extends seamlessly across modalities, and achieves a new state of the art in sparse multimodal retrieval while delivering practical benefits such as hybrid dense-sparse scoring and reduced tool-call costs in agentic search.

Dataset

The authors assemble a training dataset of 3.94 million query-document pairs drawn from three public sources.

• Echo-embedding training data: Large-scale query-document pairs covering a wide range of domains.
• MLDR training data: Provides long-document training examples.
• MMEB training sets: Multimodal data spanning diverse visual and cross-modal retrieval tasks.

Processing and usage:

• The multimodal subsets (MMEB) originally lack negative examples. To remedy this, the authors mine hard negatives using a teacher model, Qwen3-VL-Embedding-8B. For each query, the teacher retrieves the top-k most similar but non-relevant documents from the corpus, and these serve as hard negatives.
• The curated dataset, with the added hard negatives, is used to train a learned sparse retriever. No further filtering, cropping, or metadata construction details are mentioned in the provided text.

Method

The authors propose UEmbed, a unified embedding model that supports both dense and sparse retrieval within a single decoder-only architecture. As shown in the figure below:

The framework addresses the limitations of applying traditional learned sparse retrieval to causal language models by introducing a partitioned sparse head mechanism and a unified training objective.

Before detailing the architecture, it is essential to understand the foundational components. The model relies on the InfoNCE loss to maximize the similarity between a query and a positive document while minimizing it for in-batch negatives. For sparse retrieval, traditional methods like SPLADE project token hidden states onto the vocabulary and aggregate them using max-pooling with a saturating activation to produce sparse weights. However, the unidirectional attention in decoder-only models prevents effective max-pooling over all hidden states, necessitating a novel approach.

To overcome the information bottleneck of relying on a single token for sparse representation, the authors introduce a vocabulary compression and partitioning strategy. First, the vocabulary is compressed by stripping accents, lowercasing, and collapsing whitespace, merging redundant tokens to reduce the overall vocabulary size.

Next, the model appends $N$N learnable special tokens to the end of the input sequence. Under causal attention, each special token can attend to all preceding tokens, effectively summarizing the full input. To ensure each special token captures a distinct semantic subspace, the vocabulary is partitioned into $N$N disjoint subsets of approximately equal size using k-means clustering. Each special token is assigned a subset-specific sparse head to compute sparse weights for its assigned vocabulary terms via a linear projection followed by a ReLU and log transformation. The final sparse representation is formed by concatenating the subset vectors from all special tokens.

For dense retrieval, the model utilizes the hidden state of the EOS token preceding the special tokens. This design allows the dense embedding to be extracted without extra forward computation when sparse retrieval is not required.

The model is trained using a unified objective that combines losses for both retrieval modes. The total loss incorporates the InfoNCE loss for dense retrieval using cosine similarity and sparse retrieval using inner product. Additionally, FLOPS regularizers are applied to the query and document sparse weights to encourage sparsity by penalizing the squared mean term weight. Scalar coefficients balance these four loss components during training.

To train the model effectively, the authors curate a diverse dataset comprising 3.94 million samples from three publicly available sources, covering general domains, long documents, and multimodal tasks. Recognizing the importance of hard negatives for training learned sparse retrievers, especially for multimodal datasets that lack them, the authors employ a teacher model to mine hard negatives. For each query, the teacher model retrieves the top-k most similar non-relevant documents from the corpus, which are then used as hard negatives during training.

Experiment

UEmbed, scaled from 2B to 9B parameters, is evaluated on multimodal (MMEB-v2) and text (BEIR) benchmarks against a wide range of embedding baselines. The model's dense mode is competitive with the best open multimodal embedders, while its sparse mode is nearly as effective, especially on document-heavy tasks. On text retrieval, the sparse mode matches specialist models while retaining dense and multimodal capabilities. Ablations validate the causal design, joint training, and semantic vocabulary partitioning, and practical experiments show improved hybrid scoring, deployment efficiency, and agentic search performance.

UEmbed dense models achieve leading results among openly trained multimodal embedders, with the 9B variant outperforming peers like RzenEmbed-V2-7B. Sparse retrieval matches dense performance closely, with gaps under 1 point, and excels on visually-rich document tasks, making it a practical alternative. UEmbed-9B (dense) surpasses all other models trained on public data, including RzenEmbed-V2-7B and Ops-MM-Embed-7B. Sparse embeddings are on par with dense: the 9B sparse model trails by only 0.8 points, and the 4B sparse model beats the dense Ops-MM-Embed-7B. Sparse models are especially effective on VisDoc, where the 9B sparse drops only 0.1 points versus its dense counterpart. On the 4B scale, UEmbed-4B (dense) outperforms Embed-RL-4B by a clear margin, and the 2B dense model remains competitive with larger models.

UEmbed models achieve strong dense retrieval results on BEIR, with UEmbed-9B reaching the highest average nDCG@10 of 56.3, ahead of recent baselines like Qwen3-VL-Embedding-8B and GME-7B. In sparse retrieval, UEmbed-9B ties the specialist Echo-Mistral-SPLADE at 55.2, while retaining multimodal and dense capabilities in a single backbone. The models exhibit particular strength on Quora and NFCorpus. UEmbed-9B leads all dense models with an average nDCG@10 of 56.3, outperforming Qwen3-VL-Embedding-8B (55.5) and GME-7B (53.5). In sparse retrieval, UEmbed-9B matches the specialist Echo-Mistral-SPLADE at 55.2 average, demonstrating no performance compromise for its unified multimodal and dense design.

On the image subset of MMEB-v1, UEmbed outperforms a bidirectional SPLADE baseline that shares the same backbone, training data, and regularization. The dense mode achieves the largest gains on the QA task, and the sparse mode also improves across most metrics, showing that the unified causal formulation intrinsically enhances embedding quality. Dense UEmbed delivers a substantial performance increase on QA compared to the baseline, indicating better utilization of the autoregressive backbone's capabilities. Sparse UEmbed also yields consistent improvements, with the strongest boost on QA, confirming the dual-mode advantage of the causal approach.

Semantic vocabulary partitioning achieves the highest average sparse performance on the image subset, with gains in grounding and QA partially offset by a slight drop in classification. Max-distance partitioning leads in retrieval and classification, while random partitioning ranks lowest overall. Semantic partitioning improves GRD and QA by nearly a point over random, but CLS falls slightly behind. Max-distance partitioning delivers the best RET and CLS scores, outperforming the semantic method on those tasks.

Hybrid scoring, which combines dense and sparse retrieval in a single forward pass, provides modest gains for text and visually-rich documents where lexical matching adds complementary signal, but yields no improvement for natural images or video. Sparse mode alone nearly matches dense performance across modalities, making it a practical standalone option when inverted index compatibility is desired. Hybrid scoring improves text retrieval by 0.3 points and visually-rich document retrieval by 0.5 points, while image and video metrics stay essentially flat. Sparse retrieval lags dense retrieval by only small margins, with the largest gap of 2.3 points on video, and matches dense on visually-rich documents within 0.3 points.

Across multimodal retrieval benchmarks including MMEB-v1 and BEIR, UEmbed models achieve leading dense and sparse results among openly trained embedders, with sparse retrieval closely matching dense performance and excelling on visually rich documents. The unified causal formulation improves both dense and sparse embedding quality over a bidirectional SPLADE baseline, and semantic vocabulary partitioning yields the best average sparse retrieval. Hybrid dense-sparse scoring provides modest text and document gains, but sparse mode alone remains a practical standalone alternative.