One-sentence Summary

Researchers from Renmin University of China, Lenovo Group, and Wuhan University propose SpecMRL, which extends Matryoshka Representation Learning by reorganizing the feature space with a parameter-derived spectral basis ordered by feature importance for more effective prefix readout; experiments on MTEB show that SpecMRL outperforms MRL at all reduced dimensions and surpasses full-dimensional MRL using only half the embedding dimensions.

Key Contributions

• The paper introduces SpecMRL, a parametric-spectral redesign of Matryoshka Representation Learning that reorganizes the feature space using a parameter-derived spectral basis ordered by feature importance.
• The method replaces fixed nested-prefix readout with prefix readout in the learned spectral space, enabling more compact leading representations while otherwise following standard MRL.
• Experiments on MTEB show that SpecMRL outperforms MRL at all evaluated reduced dimensions, with larger gains under aggressive compression, and surpasses full-dimensional MRL using only half the embedding dimensions.

Introduction

Representation learning underpins many downstream applications, but fixed high-dimensional embeddings impose storage and latency costs. Matryoshka Representation Learning (MRL) addresses this by training nested prefixes of a full embedding in one run, yet it does not explicitly ensure that earlier dimensions carry the most useful information. Prior efforts improve low-dimensional readouts through cross-dimensional supervision, but they leave the original coordinate system unchanged, which can be poorly suited to prefix truncation. The authors propose SpecMRL, a parameter-derived spectral redesign that changes only the representation basis. It uses the Gram matrix of a learned projection to obtain an ordered orthogonal basis, and a power-iteration-based approximation avoids costly eigen solvers and unstable eigenvector gradients. Experiments on MTEB show SpecMRL outperforms MRL at reduced dimensions, gaining up to 3.80 points under 16x compression and exceeding full-capacity MRL with half the dimensions.

Method

The authors introduce SpecMRL as a drop-in modification to Matryoshka Representation Learning that replaces the fixed, heuristic ordering of nested dimensions with a learned spectral ordering. Instead of treating earlier dimensions of an embedding as inherently more important, SpecMRL encodes the latent structure of the representation space in a projection matrix $W_{\mathrm{Proj}} \in \mathbb{R}^{D \times R}$WProj​∈RD×R and derives a nested basis from that matrix.

Specifically, SpecMRL computes an ordered basis from the Gram matrix of $W_{\mathrm{Proj}}$WProj​:

where the eigenvectors are sorted according to their eigenvalues:

Given this ordering, the leading $m$m columns of $\boldsymbol{U}$U form a nested readout basis. For an input representation $\boldsymbol{X} \in \mathbb{R}^{D}$X∈RD, the spectral prefix is defined as:

This spectral prefix is then used in place of the standard nested representation in each loss term:

Because this change only affects how nested representations are constructed, the remaining Matryoshka training pipeline is preserved without additional loss functions or architectural changes.

A direct implementation of the spectral decomposition can be expensive and sensitive to ill-conditioned eigenvalue distributions. To make SpecMRL practical, the authors approximate the ordered basis using a streaming power iteration scheme. The basis is initialized from the ordered eigenvectors of the initial Gram matrix, and subsequent steps update it through one matrix multiplication followed by QR orthogonalization:

where the previous basis $\boldsymbol{U}^{(t-1)}$U(t−1) is treated as fixed while gradients propagate through the Gram matrix and QR factorization. This design avoids the cost and instability of repeatedly running full eigen decomposition during training.

At inference, the spectral basis is fixed and does not need to be recomputed from $W_{\mathrm{Proj}}$WProj​. SpecMRL therefore incurs only the cost of a $D \times D$D×D linear projection, making it efficient in practical deployment while preserving the nested representation structure required by Matryoshka-style training.

Experiment

The experiments evaluate SpecMRL against standard MRL across multiple backbones and reduced dimensions on MTEB, with ablations isolating the contributions of the parametric spectral basis, eigen decomposition, and over-parameterization. Results show that SpecMRL consistently improves low-dimensional performance and achieves its best results at only half the full dimension, while feature-derived, Gram-based, and orthogonality-only variants do not match the full method. Additional studies confirm that SpecMRL scales to larger models and longer training, remains robust to changes in nested loss weighting and sparse supervision, outperforms existing MRL improvements, and adds negligible encoding overhead with faster retrieval.

SpecMRL performs comparably to MRL at the full dimension but consistently outperforms it across all reduced dimensions on Qwen3-0.6B, Qwen2-0.5B, and Llama3.2-1B. The advantage grows as compression becomes stronger, with the largest gains at the smallest evaluated readout sizes. Across all three backbones, SpecMRL achieves its best average MTEB score at half the full dimension. SpecMRL outperforms MRL at every reduced dimension across the three evaluated backbones, while full-dimension differences remain negligible. The performance gap widens with stronger compression, showing greater gains at lower readout dimensions. SpecMRL reaches its best score using half the full dimension and supports faster retrieval with the same compact representation.

SpecMRL consistently outperforms feature-derived variants across all evaluated prefix dimensions and beta settings on MTEB. The performance gap is small at the largest prefix dimension but grows substantially as the representation dimension decreases. This supports the parameter-derived spectrum as more effective than constructing the spectral basis directly from input features. SpecMRL achieves the best performance in every comparison against FeatMRL. Feature-derived variants show only minor drops at the largest prefix dimension, but the gap widens considerably as dimension decreases. No FeatMRL beta setting matches SpecMRL, with the largest drops occurring at the smallest prefix dimension.

The hyperparameter search shows that moderate over-parameterization of the projection improves performance up to a point. The best results occur at τ=2 with a projection scale of 128D, followed closely by 256D, while average performance improves from D to 128D and then slightly decreases. Larger τ values, especially 8 and 16, lead to consistently lower performance. At τ=2, the top result is 61.61 at 128D, and the second-best result is 61.55 at 256D. Average performance across τ rises from 59.98 at D to 60.56 at 128D, then edges down to 60.51 at 256D, while τ=16 drops to the 57.8 to 58.5 range.

Across the three model families, power iteration causes only a small training throughput drop relative to MRL, while SVD and Eigh impose substantially larger overhead. The difference is especially clear on Llama3, where SVD shows the steepest degradation among all solvers. Power iteration therefore preserves near-MRL training efficiency. Power iteration has the lowest throughput degradation across Qwen3, Qwen2, and Llama3, staying below one percent relative to MRL. SVD and Eigh incur larger training overhead, with SVD showing the largest drop among the solvers on Llama3.

Ablations indicate that adding a learnable Gram-based similarity gives only limited gains over vanilla MRL and degrades quickly under stronger compression. SpecMRL maintains stronger performance as nested dimensions shrink. A learnable orthogonal basis improves over vanilla MRL but still trails full SpecMRL, suggesting spectral ordering contributes beyond orthogonality alone. SpecMRL consistently retains higher performance than both ablated variants at reduced dimensions. The Gram-based variant performs close to vanilla MRL and shows larger drops under stronger compression. The orthogonal variant improves over vanilla MRL but remains below SpecMRL, highlighting the value of learned spectral ordering.

The experiments assess SpecMRL on MTEB with Qwen3-0.6B, Qwen2-0.5B, and Llama3.2-1B, comparing it against MRL, feature-derived spectral bases, solver choices, and ablations under reduced readout dimensions. SpecMRL matches MRL at full dimension but consistently outperforms it and all feature-derived baselines under compression, with its best average scores at half the full dimension. Moderate projection over-parameterization improves results up to a point, power iteration preserves near-MRL training throughput, and SpecMRL's learned spectral ordering outperforms Gram-based and orthogonal-only variants.