One-sentence Summary

Researchers from Purdue University propose a data-centric multimodal alignment method for fine-grained food image understanding that uses target-aware data selection and VLM-based caption refinement to train complementary CLIP-style retrieval experts, then fuses their decisions through a hierarchical VLM-assisted multi-expert decision-level fusion strategy that invokes the VLM only when experts disagree, achieving more than twice the retrieval score of pure VLM-based retrieval while remaining substantially more efficient.

Key Contributions

• Target-aware data selection identifies web images close to the target distribution, and a strategically chosen 5k subset outperforms a naive 25k subset under raw web supervision.
• VLM-based caption refinement replaces noisy web captions with visually grounded, target-style descriptions, yielding an average retrieval performance gain of approximately 19%.
• A hierarchical multi-expert decision-level fusion strategy combines complementary retrieval experts and invokes a VLM only when predictions disagree, achieving more than twice the retrieval score of pure VLM-based retrieval with substantially lower VLM resource usage.

Introduction

Food image understanding demands fine-grained recognition of ingredients, cooking methods, textures, and plate composition, which is critical for dietary monitoring, nutritional analysis, and food safety. While contrastive vision-language models like CLIP can align images and descriptive text, their performance on food data is severely limited by noisy, heterogeneous web-collected training pairs that suffer from cross-modal misalignment and a domain gap between the source and target. The authors address this by prioritizing data quality over model architecture, proposing a three-stage data-centric multimodal alignment strategy: target-aware data selection to curate a compact, relevant subset of web images; VLM-based caption refinement that replaces noisy captions with visually grounded descriptions, yielding a roughly 19% performance gain; and hierarchical multi-expert decision-level fusion that combines complementary retrieval experts and invokes a VLM only when their predictions disagree, achieving more than twice the retrieval score of a pure VLM approach while substantially reducing resource usage.

Dataset

The authors construct a training dataset from web images, using target-aware selection and VLM caption refinement to bridge the gap between source web data and the target retrieval distribution.

• Dataset composition and sources: The base pool is a collection of web images (likely food images). The unlabeled test set (target images) is used solely to compute a target prototype, not for training. The final dataset consists of two curated subsets of 5k images each, with refined captions generated by a VLM.

• Subset 1: Nearest-5k

  • Size: 5,000 images.
  • Source: Ranked web images by cosine similarity to the target prototype.
  • Filtering: A CLIP image encoder computes normalized embeddings for all target images to form an average centroid (target prototype). Each web image is scored by its cosine similarity to this prototype. The top 5k images are retained.
  • Purpose: Closely aligns the training data with the target visual distribution, reducing domain shift.

• Subset 2: Random-5k

  • Size: 5,000 images.
  • Source: Uniformly sampled from the same web image pool.
  • Filtering: No target-similarity optimization; preserves broader diversity.
  • Purpose: Provides complementary expert predictions for later fusion.

• Caption refinement: For every selected image (in both subsets), the authors use the VLM Gemma 4 to generate a new English caption. The VLM is prompted with unpaired target-domain captions as in-context examples and instructed to describe only observable food details (ingredients, textures, colors, etc.). Outputs are cleaned (removing prefixes, quotation marks, formatting artifacts) to yield concise food captions.

• How the data is used: The curated image–caption pairs are used to train separate retrieval experts — one on Nearest-5k and one on Random-5k. The experts are later combined through multi-expert decision-level fusion. No cropping or additional metadata construction is detailed; the key processing is target-aware selection and caption regeneration.

Method

The authors propose a three-stage data-centric multimodal alignment method designed to reduce the source-to-target visual distribution gap and image-text semantic misalignment in web-collected training data, instantiated for food image-to-text retrieval.

Refer to the framework diagram:

In the first stage, Target-Aware Data Selection, the authors select web images that are visually closer to the test-time retrieval distribution. They summarize the target visual distribution using an image prototype. Given a fixed pretrained CLIP image encoder $g_I(\cdot)$gI​(⋅) and unlabeled test images $\{x_j^{\mathrm{tar}}\}_{j=1}^N${xjtar​}j=1N​, they compute an average normalized target prototype:

$$p_{\mathrm{tar}} = \frac{1}{N} \sum_{j=1}^{N} \frac{g_I(x_j^{\mathrm{tar}})}{\|g_I(x_j^{\mathrm{tar}})\|_2}.$$ptar​=N1​j=1∑N​∥gI​(xjtar​)∥2​gI​(xjtar​)​.

For each web image $x_i^{\mathrm{web}}$xiweb​, its similarity to the target prototype is computed as:

$$r_i = \left(\frac{g_I(x_i^{\mathrm{web}})}{\|g_I(x_i^{\mathrm{web}})\|_2}\right)^\top \frac{p_{\mathrm{tar}}}{\|p_{\mathrm{tar}}\|_2}.$$ri​=(∥gI​(xiweb​)∥2​gI​(xiweb​)​)⊤∥ptar​∥2​ptar​​.

The web images are ranked in descending order of $r_i$ri​, and the top 5k images are retained as the Nearest-5k subset. Additionally, a Random-5k subset is constructed by randomly sampling 5k images to preserve broader diversity and provide complementary signals.

In the second stage, VLM-Based Caption Refinement, the authors address noisy web captions by regenerating them using a VLM. Unpaired target captions are used as in-context examples to prompt the VLM to describe observable food details, producing curated image-caption pairs $(x_i^{\mathrm{web}}, \hat{t}_i^{\mathrm{vlm}})$(xiweb​,t^ivlm​).

In the third stage, Retrieval Expert Construction, four CLIP-style retrieval experts are trained using two backbones (DFN5B and MetaCLIP 2) on the two curated subsets (Nearest-5k and Random-5k). Each expert is trained with a symmetric InfoNCE loss.

Finally, in the Hierarchical VLM-Assisted Multi-Expert Decision-Level Fusion stage, the authors combine the predictions of the four experts. Instead of simple voting, a VLM resolves prediction disagreements. If two expert predictions agree, the shared prediction is kept; otherwise, the VLM reviews the image and the disagreement. For single-caption retrieval, the VLM selects the better matching caption. For multi-label ingredient retrieval, common ingredients are kept, and disputed ingredients are resolved via visual yes/no questions. This fusion is applied hierarchically: predictions from the two DFN5B experts are fused into one output, and predictions from the two MetaCLIP 2 experts are fused into another. These two fused outputs are then combined using the same VLM-assisted rule to obtain the final prediction.

Experiment

The evaluation on the Dishcovery challenge test set, which combines multi-ingredient recognition and caption retrieval, compares CLIP fine-tuning strategies and demonstrates that VLM-based caption refinement is critical for overcoming noisy web captions, with a compact target-aligned dataset (Nearest-5k) providing sufficient supervision. The full method hierarchically fuses four retrieval experts using a VLM for decision-level disagreement resolution, raising the official score to 0.653 and outperforming both the best single expert and direct VLM retrieval while reducing token usage dramatically. Qualitative analysis confirms that data selection and caption refinement effectively bridge the visual and semantic gaps between source and target distributions.

VLM-based caption refinement substantially improves performance across all fine-tuning methods, while nearest-neighbor data selection yields further gains when combined with refined captions. For the DFN5B backbone, last-few-layers fine-tuning with the nearest 5k samples and VLM captions achieves the best result, and a compact 5k subset suffices once captions are cleaned. On MetaCLIP 2, full fine-tuning performs comparably to LoRA, making it a simple and reproducible choice. Replacing raw web captions with VLM-generated captions consistently improves scores across all fine-tuning approaches, with large absolute gains seen in every setting. Nearest-5k data selection outperforms random sampling, and combining nearest selection with VLM captions yields the highest single-expert score. With VLM-refined captions, a 5k subset is sufficient and increasing data to 25k provides no further improvement, indicating caption quality is the primary driver. Last-few-layers fine-tuning is the most effective method for DFN5B, while WiSE-FT, which excels with raw captions, is surpassed by full and last-few-layers tuning when captions are cleaned. On MetaCLIP 2, full fine-tuning matches LoRA performance, so full fine-tuning is selected for its simplicity and reproducibility.

The full method achieves a retrieval score of 0.653, a 0.427-point improvement over pure VLM retrieval, while requiring only 6.1 million VLM tokens instead of 431.2 million. This targeted use of VLM experts yields both substantially higher accuracy and dramatically better token efficiency, with inference time also reduced by an order of magnitude despite differing hardware. The full method increases retrieval score from 0.226 to 0.653, a 0.427 absolute point gain. VLM token consumption drops from 431.2 million to 6.1 million, a reduction of over 98%.

Selecting the 5k training images most visually similar to the target distribution (Nearest-5k) with VLM-generated captions yields the best retrieval performance for both backbones, surpassing random sampling and larger 25k selections. The results indicate that a compact, target-aligned subset provides sufficient supervision, while expanding the selection to 25k offers no benefit and can even degrade performance. The Nearest-5k selection strategy improves DFN5B's score from 0.599 (random) to 0.607, while MetaCLIP 2 shows a modest gain from 0.584 to 0.587. Using a larger 25k subset with top visual similarity does not improve performance and for DFN5B leads to a drop to 0.581, suggesting that a small, curated dataset is sufficient. Targeted visual similarity selection benefits DFN5B more than MetaCLIP 2, where random sampling already performs competitively.

VLM-generated captions consistently improve fine-tuning performance, and selecting the 5k training images most visually similar to the target distribution yields the best results when combined with those refined captions. For the DFN5B model, last-few-layers fine-tuning on this compact subset is most effective, while full fine-tuning is adopted for MetaCLIP 2 because it matches LoRA and is simpler to reproduce. The full method attains a large retrieval accuracy gain while consuming over 98% fewer VLM tokens, confirming that caption quality and targeted data curation are the primary drivers, and that a small, curated dataset suffices without benefiting from larger data.