One-sentence Summary

Shanghai Jiao Tong University, Zhongguancun Academy, and Shanghai Innovation Institute introduce CLBench-V, a multimodal context-learning benchmark that organizes tasks around context grounding, new information application, and new knowledge learning, combining converted public and automatically constructed datasets spanning science, finance, long-document understanding, spatial reasoning, and web-based visual question answering, and across 3,443 instances and six recent multimodal models the best overall score is only 0.2847, underscoring the unsolved challenges in multimodal context learning.

Key Contributions

• CLBench-V introduces a diagnostic benchmark for multimodal context learning with 3,443 instances spanning science, finance, long-document understanding, spatial reasoning, and web-based visual question answering, combining converted public datasets with newly constructed tasks built through automated construction and filtering.
• The benchmark defines a three-level hierarchy (context grounding, new information application, and new knowledge learning) that isolates whether failures stem from missing evidence, misuse of context-specific information, or inability to acquire and transfer context-defined knowledge.
• Evaluation of six recent multimodal models yields a best overall score of 0.2847, revealing that multimodal context learning is far from saturated, and per-dimension breakdowns show InternVL3.5-30B-A3B leads on context grounding and new knowledge learning while Qwen3.5-Plus performs best on new information application, with further analysis of judge reliability, input complexity, and failure cases.

Introduction

Large language models increasingly use context provided at inference time to solve problems, extracting task-specific facts, rules, or patterns rather than relying solely on parametric knowledge. This context-learning capability is critical for real-world deployments where models encounter unseen documents and dynamic environments, and where relevant evidence often appears across multimodal formats like text, figures, tables, charts, and layouts. Existing benchmarks for context learning are largely unimodal and textual, leaving a gap in evaluating how well models can locate, apply, and acquire new knowledge from multimodal contexts. Prior long-context studies show that even models with large input windows may fail to internalize context-defined rules, resolve cross-modal conflicts, or transfer newly provided information, but they often lack diagnostic accuracy to pinpoint where context use breaks down. The authors introduce CLBench-V, a multimodal context-learning benchmark organized around a three-level hierarchy: L0 context grounding, L1 new information application, and L2 new knowledge learning. This design isolates failures in locating multimodal evidence, misusing accessed context, or ignoring context-defined knowledge in favor of prior knowledge. The benchmark combines converted public datasets with newly constructed domain-specific tasks, such as medical-paper conclusion inference and financial-report ROE analysis, and evaluates six multimodal models across 3,443 instances under a unified framework.

Dataset

The authors build CLBench-V as a multimodal evaluation benchmark from two complementary sources, all unified under a single schema and organized by capability: context grounding, new information application, and new knowledge learning.

• Data Integration (public benchmarks)

  • Source benchmarks are selected to cover visual fact verification, long-document understanding, scientific inconsistency reasoning, high-resolution search, multi-image reasoning, and context-defined rule following (e.g., Pix2Fact, MMLongBench, PRISMM-Bench, ReasonMap, InSight-o3, MIRBench, ZeroBench, CourtSI-Bench, BrowseComp-V, CL-Bench).
  • Four admission criteria are enforced:
    • Visual dependency: answers cannot be derived from text-only shortcuts.
    • Low irrelevant context: tasks remain focused on information application and knowledge learning, not pure needle-search.
    • Multi-hop preference: samples require compositional reasoning across multiple evidence pieces.
    • Scope exclusion: video and quasi-video tasks are excluded to keep focus on static multimodal context learning.
  • Filtering uses Qwen3.5-Plus as an automatic inspector to reject shortcut-solvable instances, followed by manual review for quality, diversity, and alignment with the three capability axes.
  • Retained samples are normalized: options are embedded for MCQ items, Markdown and image-rendered document variants are preserved when available, topology fields are kept for route-structure validation, and external search evidence is moved into the context. This pipeline produces the integrated public-benchmark portion.

• Data Construction (new domain-specific tasks)

  • Two tasks are built to cover gaps in existing benchmarks while sharing the same schema and evaluator interface.
  • Financial report ROE analysis: recent financial reports are converted from PDF to a Markdown-style multimodal representation. The model must perform DuPont decomposition (compute ROE from net profit margin, asset turnover, equity multiplier) using values distributed across pages and visual structures. This primarily evaluates new information application.
  • Paper conclusion inference: recent medical papers are converted to multimodal Markdown. Two truncated versions are created: a reference-construction version that includes content up to and including the Result section (used to generate reference findings), and an evaluation version that stops before the Result section. At test time, the model receives only the evaluation version and must infer scientific findings that would be supported by the paper’s results. After filtering, 581 clean examples are retained. Scoring penalizes over-prediction with a max(0, (hits - extras) / ref_count) formula.

• How the data is used

  • The benchmark is used entirely for evaluation. There is no training split; all tasks are zero-shot or few-shot evaluation settings.
  • Task outputs are scored with deterministic evaluators for structured answers and judge-based semantic evaluation for open-ended outputs.
  • The unified schema supports direct comparison across the three capability dimensions, isolating context learning from shortcut retrieval.

Method

The authors design CLBench-V to evaluate multimodal large language models' ability to learn from provided multimodal context and apply that knowledge to solve tasks in realistic scenarios. The benchmark is structured around a unified conversion, inference, and scoring framework that supports cross-level comparability and fine-grained diagnosis of failure modes.

Refer to the framework diagram.

To make this ability measurable and diagnostic, the authors organize multimodal context learning into a three-level capability hierarchy. This hierarchy isolates whether failures stem from missing evidence, misusing context-specific information, or failing to acquire context-defined knowledge.

• L0: Context Grounding: At this level, the model must locate, read, and combine evidence from newly provided multimodal context without needing to learn a new rule system. Tasks at this level test evidence access and grounding fidelity, including visual search, counting, spatial localization, and cross-image binding.
• L1: New Information Application: Here, the model must use new facts, values, or document-specific information from the context while relying on its existing reasoning skills. These tasks test whether the model can correctly bind context-specific values and use them in existing reasoning procedures, avoiding outdated prior knowledge.
• L2: New Knowledge Learning: This is the most challenging level, requiring the model to acquire new rules, empirical patterns, or scientific conclusions from the context and apply them. The model must synthesize context-defined knowledge and transfer it to answer generation.

The benchmark is built from two complementary sources: integrated public benchmarks and newly constructed domain-specific tasks. Both sources are mapped into a normalized schema.

For data integration, the authors apply explicit curation criteria to public benchmarks to ensure they evaluate context learning rather than shortcut retrieval. These criteria include visual dependency, low irrelevant context, multi-hop preference, and scope exclusion. Candidate instances undergo conversion-time rejection sampling and manual review to retain high-quality samples that satisfy the criteria.

For data construction, the authors create domain-specific tasks to target capability gaps.

• Financial report ROE analysis: Models perform DuPont-style reasoning to compute ROE from converted financial reports. The task requires identifying and combining quantities such as net income, revenue, total assets, and shareholders' equity. The standard three-factor DuPont decomposition is used:

• Paper conclusion inference: Models infer paper-specific scientific findings from recent medical papers. The PDF is converted into a Markdown-style multimodal representation. The evaluation version includes content before the Result section, requiring the model to synthesize experimental motivation, methods, and multimodal evidence to predict scientific findings. The paper-level score is calculated as:

where $N_{\mathrm{ref}}$Nref​ is the number of reference findings, $N_{\mathrm{hit}}$Nhit​ is the number of correct model-predicted findings, and $N_{\mathrm{extra}}$Nextra​ is the number of unsupported extra findings. This subtraction term penalizes over-prediction.

All datasets are evaluated in a unified framework with task-specific metrics, using deterministic evaluators for structured outputs and judge-based semantic evaluation for open-ended outputs.

Experiment

CLBench-V evaluates multimodal models on a three-level hierarchy of context learning, ranging from grounding evidence in the provided input (L0) to applying new information (L1) and learning new knowledge (L2). Overall model performance remains low, with the hardest L2 tasks requiring synthesis and transfer of context-defined knowledge, and no single model dominates across all levels. The benchmark reveals distinct capability profiles and failure modes such as evidence misbinding and prior override, while simple measures of input length and image count do not strongly predict performance.

The CLBench-V benchmark organizes multimodal context-learning tasks by capability level, with the L0 level comprising 1,074 instances across six datasets that test context grounding via subway map reasoning, visual search, and sports spatial intelligence. Each task uses deterministic evaluators for structured outputs, while open-ended outputs rely on judge-based semantic evaluation, where judge selection can measurably influence reported scores. The L0 level aggregates 1,074 samples from six datasets spanning subway map route planning, high-resolution visual search, hard visual QA, and sports spatial intelligence. Judge-based semantic evaluation for open-ended tasks introduces non-negligible score variation, making transparent reporting of the judge model essential for reliable comparisons.

The best model, InternVL3.5-30B-A3B, reaches an overall score of only 0.2847, indicating that multimodal context learning remains challenging and the benchmark is not saturated. Capability profiles differ sharply: InternVL3.5-30B-A3B achieves the top L0 grounding and L2 knowledge scores but is the weakest on L1 information, while Qwen3.5-Plus and Qwen3.6-27B lead on L1 yet lag behind on L2. These contrasting strengths, together with model-dependent sensitivity to additional images, reveal that current systems fail in complementary ways. InternVL3.5-30B-A3B leads overall and on L0 grounding and L2 knowledge, but its L1 information score (0.1313) is lower than every other model, indicating difficulty applying small, document-specific facts. Qwen3.5-Plus and Qwen3.6-27B share the strongest L1 information scores (0.2954 and 0.2878), yet their L2 knowledge scores are substantially lower, showing a trade-off between using newly supplied information and acquiring context-defined knowledge. Kimi-K2.6 ranks second overall and second on L2 knowledge, while GPT-5.4 is second on L0 grounding but ranks lowest on L2, highlighting that no single model family excels across all context learning levels. Additional images improve InternVL3.5-30B-A3B's ranking (especially after excluding extremely long financial reports), whereas Qwen3.6-27B's rank does not benefit, suggesting that extra views act as richer evidence for one model and as visual distraction for the other. Overall performance remains low for all models; input length alone does not explain failures, and the main obstacles stem from usable context limits and model-specific handling of visual evidence.

Multimodal context learning remains difficult for current models, with no single system excelling across all capability levels. InternVL3.5-30B-A3B leads on document-level knowledge tasks like paper conclusion inference but fails on long financial reports due to context limits, while Qwen3.6-27B and Qwen3.5-Plus are stronger at grounding and information extraction respectively. InternVL3.5-30B-A3B obtains the highest score on the Paper dataset but scores zero on Financial Report ROE, exceeding its maximum context length. Qwen3.6-27B achieves the strongest L0 Grounding result by a large margin. Kimi-K2.6 substantially outperforms others on L0-L1 boundary tasks. Qwen3.5-Plus attains the best L1-L2 boundary score, reflecting superior application of supplied information. Doubao-Seed-2.0-Lite and GPT-5.4 tie for the highest L2 Knowledge aggregate score. InternVL3.5-30B-A3B earns the top L0-L1 boundary score of 0.0680, well below Kimi-K2.6's 0.6800, showing its difficulty with cross-level grounding.

Different LLM judges produce notably different scores on the same multimodal tasks, with the average score varying by over 0.05 across five judge models. The smallest judge, Qwen3-VL-4B-Instruct, assigned the highest average score, while the largest, Qwen3-VL-32B-Instruct, assigned the lowest. This underscores the need for transparent judge reporting and calibration. Judge choice causes non-negligible score variation; average scores span from 0.1564 to 0.2122. Qwen3-VL-4B-Instruct, the smallest model, gives the highest average, while Qwen3-VL-32B-Instruct gives the lowest.

The CLBench-V evaluation uses six L0 datasets to test multimodal context learning with both deterministic and judge-based semantic scoring, where judge model choice measurably shifts reported scores. Overall performance is low across all models (best aggregate 0.2847), and capability profiles diverge sharply: InternVL3.5‑30B‑A3B leads on grounding and knowledge but almost fails on information extraction, while Qwen variants achieve the highest information scores yet lag on knowledge, and extra images act as helpful evidence for one model but visual distraction for another. Separate judge experiments show that smaller judge models assign notably higher average scores than larger ones, underscoring the need for transparent judge reporting and calibration.