One-sentence Summary

Researchers from Carnegie Mellon University, Institut Polytechnique de Paris, University of Michigan, and other institutions propose HarnessEval-W, an agentified evaluation pipeline that adapts the LLM harness paradigm to world model benchmarking by decomposing each evaluation case into measurable subproblems and spawning specialized sub-agents that build transparent, verifiable evidence trees, with judgments aligning closely with human preferences across 18 world models and 330 evaluation cases.

Key Contributions

• The paper introduces HarnessEval-W, an agentified evaluation pipeline that adapts the LLM harness paradigm to world model benchmarking by decomposing evaluation cases into measurable subproblems, assigning specialized sub-agents, and validating their evidence into a transparent reasoning tree.
• The method unifies evaluation of observation quality, transition correctness, and world persistence across 330 cases and 18 world models, where its judgments align closely with human preferences and produce fine-grained, verifiable rollout diagnoses.
• The work open-sources the pipeline as a live, extensible benchmark and outlines directions for test-time scaling, growing skill libraries, and recursively self-improving evaluation capabilities.

Introduction

World models are increasingly evaluated on generated video, but assessing physical causality, geometric consistency, and observation realism remains fragile. Existing benchmarks typically rely on fixed rubrics or static question-answer probes, producing scores that are difficult to explain, verify, or trace to specific model failures. The authors introduce HarnessEval-W, an agentic evaluation harness that formalizes human evaluation as a hierarchical pipeline: a parent agent interprets each case, routes it to applicable skills, spawns specialized sub-agents with diagnostic tools, and merges validated evidence into a transparent reasoning tree. HarnessEval-W covers 330 cases across Observation Quality, Transition Correctness, and World Persistence, evaluates 18 world models, and shows that its judgments align with human preferences while making each score auditable.

Dataset

The authors construct HarnessEval-W as a 330-case benchmark for evaluating world models. Each case includes an initial image, an action specification with text instruction, camera trajectory, control sequence, rollout plan, and physical parameter conditions, plus an expected outcome tied to a probe family.

Sources and construction

• World metadata is sampled from a structured scene taxonomy with six axes: Environment, Foreground, Midground, Scene Density, Appearance, and Perspective.
• Each case is also assigned one of six probe families: Exploratory Transition, Intentional Transition, Physical Transition, Drift Resistance, Revisit Consistency, and Offscreen Evolution.
• The sampler selects compatible taxonomy combinations and rejects cases whose entities or spatial layouts cannot support the selected interaction.
• An image generator creates the initial observation from a structured prompt built from the sampled metadata.
• An image-grounded planner produces the concrete action and rollout plan from the image, without changing the probe family or adding entities absent from the image.
• A case validator audits each image-action pair for target visibility, action feasibility, outcome specificity, and evidence adequacy. Cases with unclear return paths or hidden processes, such as revisit and offscreen cases, are resampled or regenerated.

Composition and filtering

• The released benchmark contains 330 cases spanning diverse environments, foreground entities, spatial layouts, scene densities, visual appearances, and perspectives.
• The six probe families divide into transition-oriented cases, covering short state, physical, and exploratory interventions, and persistence-oriented cases, covering longer rollouts for drift resistance, revisit consistency, and offscreen evolution.
• Observation Quality is evaluated in every case; the other families correspond to Transition Correctness and World Persistence settings.
• The authors use validation as a filtering gate: candidates that fail the validity audit are returned to the sampler rather than retained.

Usage

• HarnessEval-W is used as an evaluation benchmark for world models, probing both immediate world-state transitions and consistency of world dynamics over time.
• The paper does not describe a training split or mixture ratio for this benchmark; it is presented as an evaluation set with intervention and rollout cases.

Method

The authors formulate an interactive world model as predicting future observations conditioned on historical observations and user-specified actions. Given initial observations $\{ o _ { i } \} _ { i = - T } ^ { 0 }${oi​}i=−T0​ and future actions $\{ a _ { i } \} _ { i = 0 } ^ { t - 1 }${ai​}i=0t−1​, the distribution of future observations $\{ o _ { i } \} _ { i = 1 } ^ { t }${oi​}i=1t​ is decomposed leveraging the hidden states $\{ s _ { i } \} _ { i = 0 } ^ { t }${si​}i=0t​ of the world model:

$$P \left(o _ {1}, \dots, o _ {t} \mid o _ {- T}, \dots, o _ {0}; a _ {0}, \dots, a _ {t - 1}\right) \propto P \left(s _ {0} \mid o _ {- T}, \dots, o _ {0}\right) \prod_ {i = 1} ^ {t} S \left(o _ {i} \mid s _ {i}\right) T \left(s _ {i} \mid s _ {i - 1}, a _ {i - 1}\right)$$P(o1​,…,ot​∣o−T​,…,o0​;a0​,…,at−1​)∝P(s0​∣o−T​,…,o0​)i=1∏t​S(oi​∣si​)T(si​∣si−1​,ai−1​)

This factorization exposes three fundamental abilities: rendering observations from the current state ($S$S), updating the state under actions ($T$T), and maintaining a coherent state sequence over time. Consequently, the authors construct three corresponding evaluation axes. Observation Quality assesses whether the rendered observation is visually reliable, covering perceptual quality and temporal coherence. Transition Correctness focuses on whether the state transition faithfully executes requested actions, evaluating exploratory, intentional, and physical transitions. World Persistence evaluates whether the sequence of predicted states remains coherent as the world evolves, testing drift resistance, revisit consistency, and offscreen evolution.

To evaluate these axes in a highly context-dependent manner, the authors design a hierarchical agentic workflow that generates a reasoning trace grounded in the context of each case.

The evaluation begins with case-specific skill routing. HarnessEval-W interprets the case context, including the initial image, action prompt, and evaluation setting, and routes the case to one or more reusable skills drawn from a predefined skill library. This ensures that the system asks the right high-level questions for each unique world. Once a high-level skill is assigned, the system performs a granular evaluation via sub-agents rather than issuing a single holistic evaluation.

For each high-level skill, the evaluation is decomposed into a set of sub-questions answered by specialized sub-agents. Taking the Intentional Change Verifier as an example, the skill decomposes its evaluation into eight measurable sub-questions such as target visibility, transition occurrence, and final state validity. Before querying these specific sub-agents, an extra sub-agent predicts and reasons about the expected outcome from the case context. Guided by this specification, parallel sub-agents inspect the rollout and return discrete scores with diagnostics. The parent skill agent then aggregates these outputs into a structured evidence tree and produces the final case score, ensuring the reasoning trace records both the final answer and the supporting evidence.

To probe the limits of world models with diverse and realistic evaluation cases, the authors design an agentic case construction pipeline.

The pipeline first samples an initial world setup from a predefined scene taxonomy and probe family. The scene taxonomy provides complete descriptions for world initialization across six complementary axes, including environment, foreground, midground, scene density, appearance, and perspective. The probe family specifies the interaction type and expected evidence, such as exploratory transition or drift resistance. With this metadata, a series of agents handles case authoring. An Image Generator converts the metadata into a structured prompt to create the initial observation. An Image-grounded Planner then specifies concrete actions, including text instructions and camera trajectories, ensuring the action is meaningful and supported by observable evidence. Finally, a Case Validator audits the proposed image-action pair to verify target visibility, action feasibility, and outcome specificity. Candidates that fail the validity gate are returned to the sampler for resampling, concentrating expensive reasoning on ambiguous candidates while automating the majority of the construction process.

Experiment

The experiments benchmark 18 interactive world models on 330 cases across eight metrics covering observation quality, transition correctness, and world persistence, finding that text-driven general-purpose generators such as Seedance 2.0 and Wan 2.7 rank highest while different models lead on different axes. HarnessEval-W itself is validated against human pairwise judgments, outperforming WBench in alignment and remaining stable across repeated evaluations. Further analysis shows that intentional and physical transition scores are strongly correlated, exploratory transition is largely independent, and fine-tuning text-to-video models into action-conditioned models tends to improve revisit consistency but reduces physical and intentional interaction performance.

HarnessEval-W organizes evaluation into three axes derived from world-model factorization: observation quality, transition correctness, and world persistence. These axes decompose into eight detailed settings, each tied to a core world-state question about rendering, action execution, or long-term coherence. Observation quality is treated as a foundation and evaluated in every case, while transition and persistence settings define targeted probe families. Observation quality covers render quality and physical plausibility, and it is assessed across all cases rather than forming a separate probe family. Transition correctness distinguishes exploratory, intentional, and physical transitions, checking whether viewpoint changes, targeted edits, or physical interventions occur as requested. World persistence includes drift resistance, revisit consistency, and offscreen evolution, requiring stable properties to stay invariant while dynamic properties continue evolving consistently.

Overall scores are led by text-conditioned prompt image-to-video models, with Seedance 2.0, Wan 2.7, Kling 3.0, and MiniMax H3 closely grouped at the top. Strength varies by evaluation axis: Wan 2.7 leads intentional transition and physical transition correctness, while Seedance 2.0, HY-WorldPlay 1.5, and SANA-WM lead drift resistance, revisit consistency, and offscreen evolution, respectively. The paper attributes the top text-conditioned models' advantage to large-scale training that improves action-consequence prediction. Text-conditioned prompt image-to-video models occupy the top of the leaderboard, with the four highest overall scores all using the Prompt I2V interface. Overall performance is tightly clustered among the leading models, but no model leads every metric; different models specialize in different skills. Wan 2.7 has relatively lower persistence scores but ranks first in intentional transition and physical transition correctness. Seedance 2.0 leads drift resistance, HY-WorldPlay leads revisit consistency, and SANA-WM leads offscreen evolution.

HarnessEval-W evaluates world-model capabilities along three axes derived from world-model factorization: observation quality, transition correctness, and world persistence, with observation quality treated as a foundation across all cases and the other axes divided into targeted probe families such as exploratory, intentional, and physical transitions as well as drift resistance, revisit consistency, and offscreen evolution. The results show that text-conditioned prompt image-to-video models lead overall, with Seedance 2.0, Wan 2.7, Kling 3.0, and MiniMax H3 closely grouped, while performance varies by axis; for example, Wan 2.7 is strongest on intentional and physical transition correctness, whereas Seedance 2.0, HY-WorldPlay 1.5, and SANA-WM lead drift resistance, revisit consistency, and offscreen evolution respectively. Overall, the paper attributes the top models' advantage to large-scale training that improves action-consequence prediction, and concludes that no single model dominates every metric, with different models specializing in different skills.