One-sentence Summary

HKUST (Guangzhou), Tencent Yuanbao, and the University of Hong Kong propose EviRank, a training-free multimodal re-ranker that casts ranking as semantic constraint satisfaction by parsing queries into evidence packages with six semantic slots each marked required, forbidden, or ignorable, combining deterministic rubric scoring with evidence-grounded listwise comparison to achieve state-of-the-art across five retrieval benchmarks, and optionally distilling a lightweight student that retains over 90% of the teacher's capability at substantially lower cost.

Key Contributions

• EviRank recasts multimodal image re-ranking as a semantic constraint satisfaction problem, parsing text-only, image-only, or composed queries into unified typed evidence across six semantic slots with required, forbidden, or ignorable labels.
• The training-free, evidence-conditioned procedure combines deterministic rubric scoring with evidence-grounded listwise comparison, and analysis shows that structured evidence is robust, interpretable, and that forbidden constraints improve fine-grained discrimination.
• Across five benchmarks spanning text-to-image, image-to-image, and composed image retrieval, EviRank achieves state-of-the-art performance, and the structured evidence serves as supervision for distilling a compact student that retains over 90% of the teacher’s capability at substantially lower cost.

Introduction

Real-world image search queries often combine text and image cues and impose multiple fine-grained constraints at once, such as preserving an entity while modifying a specific attribute and ignoring incidental factors like background or lighting. This matters because relevance depends on verifying whether a candidate image satisfies those semantic constraints, not simply on global similarity. Prior multimodal re-rankers either collapse all query dimensions into a single dense representation that cannot expose fine-grained constraint violations, or rely on free-form chain-of-thought reasoning that can omit implicit constraints, hallucinate evidence, and cover semantic dimensions inconsistently. The authors formulate multimodal image re-ranking as evidence-conditioned semantic verification and introduce EviRank, which parses queries into typed Evidence Frames with required, forbidden, and ignorable statements over six semantic slots, then mines and verifies that evidence in a training-free two-stage procedure and uses the structured signals to distill a compact student re-ranker.

Method

The authors present EviRank, a framework that recasts multimodal image re-ranking as semantic constraint satisfaction. Rather than relying on simple similarity scores, the system parses a query into a unified package of structured evidence criteria and re-ranks candidates by evaluating which criteria they satisfy or violate.

Refer to the framework diagram below to understand the complete pipeline.

The core of this approach is the evidence representation. Regardless of whether the input query is text-only, image-only, or composed, it is mapped to a modality-agnostic evidence package $\mathcal{E}(q) = (g, \{(R_s, F_s, I_s)\}_{s \in \mathcal{S}})$E(q)=(g,{(Rs​,Fs​,Is​)}s∈S​). Here, $g$g is a concise global summary of the query intent, and $\mathcal{S}$S represents a default schema of six semantic slots: ENTITIES, ATTRIBUTES, ACTIONS, RELATIONS, SCENE, and KEYDETAILS. For each slot, the system extracts three lists of short, checkable sentences: required constraints $R_s$Rs​ that must hold, forbidden constraints $F_s$Fs​ that must not hold, and ignorable constraints $I_s$Is​ that should not influence the decision.

To build this evidence package, the authors employ a training-free query-to-evidence mining process using a single multimodal large language model (MLLM) teacher. First, the query undergoes normalization to expose its semantic content uniformly. Text queries are verbalized to make implicit visual details explicit, image queries are captioned, and composed queries are mapped by treating the reference image as preserved constraints and the modification text as updated or forbidden constraints. Following normalization, the MLLM emits the full evidence package in a single structured pass, generating positive statements for all constraint types to ensure consistent matching.

Once the evidence is mined, the framework performs evidence-conditioned verification and re-ranking through a two-stage process. The first stage is deterministic rubric scoring, which evaluates candidates independently. Let $m(\cdot, c)$m(⋅,c) be a matching operator measuring whether a constraint sentence holds for a candidate $c$c. The slot satisfaction and violation rates are calculated as:

$$\text{Match}_s(c) = \frac{1}{|R_s|} \sum_{r \in R_s} m(r, c)$$Matchs​(c)=∣Rs​∣1​r∈Rs​∑​m(r,c) $$\text{Viol}_s(c) = \frac{1}{|F_s|} \sum_{f \in F_s} m(f, c)$$Viols​(c)=∣Fs​∣1​f∈Fs​∑​m(f,c)

The rubric score aggregates these rates with a forbidden penalty and an optional cross-modal consistency term:

$$S_{\text{rub}}(c) = \sum_{s \in \mathcal{S}} w_s (\text{Match}_s(c) - \beta \text{Viol}_s(c)) + \gamma \text{Cons}(c)$$Srub​(c)=s∈S∑​ws​(Matchs​(c)−βViols​(c))+γCons(c)

where $\beta$β controls the penalty strength and $\text{Cons}(c)$Cons(c) measures alignment for composed queries. Ignorable constraints act as an invariance mask, preventing non-discriminative variations from dominating the score.

To resolve close calls between visually similar candidates, the second stage applies evidence-grounded listwise refinement. The MLLM teacher is prompted with the query, the global intent, the required and forbidden constraints, and the top candidates jointly. It returns a refined permutation where satisfied required constraints raise the rank and salient forbidden mismatches are heavily penalized. In practice, candidates are sorted by the rubric score, and the teacher only verifies and locally reorders the top $M$M candidates to reduce computational cost.

Beyond inference, the structured evidence package serves as decomposable supervision for distilling a lightweight student re-ranker. During training, the student receives a query-conditioned supervision bundle comprising the evidence package, per-candidate slot-wise satisfaction rates, calibrated listwise scores $\mathbf{r}$r, a self-assessed confidence $p$p, and a set of hard pairs $\mathcal{H}$H. Crucially, at inference time, the student consumes only the raw query and candidate image, predicting a relevance score $s_i$si​ without requiring the teacher MLLM or evidence cache.

The distillation process optimizes three objectives that mirror the structured components of the supervision bundle. Score distillation matches the student scores $\mathbf{s}$s to the standardized teacher scores $\hat{\mathbf{r}}$r^ using a KL divergence loss:

$$\mathcal{L}_{\text{score}} = \text{KL}(\text{softmax}(\hat{\mathbf{r}}/\tau) \| \text{softmax}(\mathbf{s}/\tau))$$Lscore​=KL(softmax(r^/τ)∥softmax(s/τ))

Hard-pair supervision imposes a margin $\delta$δ on teacher-flagged pairs, weighted inversely to the teacher score gap to focus on closely-scored competitors:

$$\mathcal{L}_{\text{pair}} = \sum_{(i, j) \in \mathcal{H}} w_{ij} \max(0, \delta - (s_i - s_j))$$Lpair​=(i,j)∈H∑​wij​max(0,δ−(si​−sj​))

where $w_{ij} = 1 / (|r_i - r_j| + \epsilon)$wij​=1/(∣ri​−rj​∣+ϵ). Optionally, slot-wise supervision trains a lightweight auxiliary head via binary cross-entropy to predict slot-wise labels $y_{s,i}$ys,i​:

$$\mathcal{L}_{\text{slot}} = -\frac{1}{K|\mathcal{S}|} \sum_{i, s} [y_{s,i} \log \hat{y}_{s,i} + (1 - y_{s,i}) \log(1 - \hat{y}_{s,i})]$$Lslot​=−K∣S∣1​i,s∑​[ys,i​logy^​s,i​+(1−ys,i​)log(1−y^​s,i​)]

The final training loss combines these terms, modulated by the teacher confidence $p$p to down-weight uncertain supervision:

$$\mathcal{L} = p \cdot (\mathcal{L}_{\text{score}} + \lambda \mathcal{L}_{\text{pair}} + \eta \mathcal{L}_{\text{slot}})$$L=p⋅(Lscore​+λLpair​+ηLslot​)

The auxiliary head is discarded after training, ensuring the deployed student model maintains a low inference footprint.

Experiment

EviRank is evaluated across text-to-image, image-to-image, and composed retrieval benchmarks, combining structured evidence extraction with listwise re-ranking. The results demonstrate that rubric-based evidence and listwise comparison are complementary, with each slot type (Required, Forbidden, Ignore) contributing to fine-grained disambiguation, while evidence extraction remains stable across teachers and prompt perturbations. The distilled student inherits this reasoning efficiently, and overall gains stem from the structured evidence schema rather than prompt engineering.

EviRank is the only method that provides a fixed, reusable slot taxonomy with typed required, forbidden, and ignorable evidence, enabling deterministic auditable scoring and a teacher-free distilled student. The structured schema yields stable rankings across repeated runs, prompt variations, and different teacher models, with most re-ranking gains coming from the slot-wise formulation rather than prompt phrasing. The six evidence slots are near-orthogonal and complementary, capturing distinct semantic dimensions. Only EviRank defines a fixed, reusable slot taxonomy with explicit FORBIDDEN and IGNORABLE evidence types, making deterministic per-slot scoring and auditing possible. Structured evidence extraction maintains high stability: Kendall's τ ≥ 0.89 and Top-1 agreement ≥ 91% across repeated runs, prompt perturbations, and different teachers. Slot-wise R/F/I formulation accounts for the majority of re-ranking improvement (+6.5 R@1 on COCO), while free-form reasoning adds only marginal gains. The six evidence slots show low pairwise similarity (mean 0.18), confirming they capture complementary rather than redundant semantic aspects. A distilled student model achieves efficient re-ranking (~800ms per query) with large gains over CLIP retrieval, without needing the teacher or evidence cache at runtime.

On Flickr30k, EviRank variants improve retrieval over the no-rerank baseline and prior reranking methods, with EviRank-plus delivering the strongest listed R@1 and MRR@5. The rubric-only EviRank-mini remains competitive without test-time MLLM inference, while full EviRank and EviRank-plus add further gains. In related reported results, EviRank-pro with a BLIP-2 backbone reaches 95.61% R@1, surpassing CoTMR by 6.32 points. EviRank-plus achieves the best listed Flickr30k retrieval performance, lifting R@1 and MRR@5 above all compared methods. Even rubric-only EviRank-mini outperforms prior methods that do not use MLLM inference at test time, while stronger EviRank variants extend the lead. With a BLIP-2 backbone, EviRank-pro attains 95.61% R@1 on Flickr30k, a 6.32-point improvement over CoTMR.

On COCO with CLIP-ViT-B/32, EviRank variants consistently outperform the no-rerank baseline and prior reranking methods. EviRank-pro reaches 64.6% R@1, a 6.3-point improvement over CoTRR, while even the rubric-only EviRank-mini surpasses CoTRR without test-time MLLM inference, demonstrating that structured evidence alone provides a strong reranking signal. EviRank-pro achieves 64.6% R@1, improving by 25.2 points over the no-rerank baseline and by 6.3 points over CoTRR. EviRank-mini, which uses only a fixed rubric and no MLLM at test time, reaches 59.9% R@1, outperforming CoTRR (58.3%).

On SoP and CUB-200 with DINOv2, reranking and query expansion methods improve retrieval mAP over the no-rerank baseline, with LoCoRE-small achieving the strongest mAP among reported baselines. Recall-oriented metrics on CUB-200 show larger gains at deeper cutoffs for several rerankers, while R@1 improvements are comparatively modest for the listed baselines. The cited EviRank-pro results reach higher R@1 on both datasets, surpassing LoCoRE-base by notable margins. LoCoRE-small records the best R@1 and mAP on both datasets among the reported baseline methods. All reranking or query expansion methods improve mAP over no rerank on SoP and CUB-200. On CUB-200, deeper recall cutoffs such as R@4 improve more than R@1 for several baselines. EviRank-pro is reported to exceed LoCoRE-base by clear margins on SoP and CUB-200 R@1.

On FashionIQ, EviRank achieves the highest retrieval accuracy across all categories, with the largest R@10 gains on Toptee and Dress compared to prior methods. The rubric-only EviRank-mini already surpasses previous baselines without test-time MLLM calls, while the full EviRank adds further improvements, demonstrating that structured evidence and listwise reasoning are complementary. Stability experiments confirm that evidence extraction remains robust under repeated runs, prompt variations, and teacher substitutions, with only a moderate accuracy drop when using a cheaper teacher. EviRank achieves the best R@10 on all three FashionIQ categories, with a 7.1-point improvement on Toptee over ImageScope. EviRank-mini, which relies solely on rubric-based evidence, outperforms ImageScope, showing the value of structured evidence without MLLM inference. Replacing the teacher with a cheaper model reduces Toptee R@10 from 42.9 to 38.3 but preserves high ranking stability (Kendall's τ=0.90, Top-1 agreement 91.1%). The gap between EviRank-mini and EviRank confirms that listwise MLLM reasoning provides complementary gains beyond structured evidence alone.

Experiments on image-text retrieval benchmarks including Flickr30k, COCO, SoP, CUB-200, and FashionIQ show that EviRank’s fixed slot taxonomy with required, forbidden, and ignorable evidence types captures complementary semantic dimensions, with the structured slot-wise formulation delivering the majority of re-ranking gains and a rubric-only variant already outperforming many baselines without test-time MLLM inference. The evidence extraction proves highly stable across repeated runs, prompt changes, and teacher substitutions, and a distilled student model enables efficient re-ranking without runtime access to the teacher or evidence cache. These findings confirm that deterministic, auditable slot-based reasoning, combined with listwise MLLM scoring, consistently advances retrieval accuracy over prior methods.