One-sentence Summary

Researchers from Google, the University of Illinois at Urbana-Champaign, and the University of California, Los Angeles introduce VIS-Ground, which performs Structured Context Abstraction, Generation Constraints Induction, and Constrained Video Generation to transform heterogeneous context into executable constraints that jointly handle source, rendered history, and viewer intent, achieving an average absolute improvement of 10.3 points over the strongest per-backbone baselines across three video generation backbones.

Key Contributions

• The paper formalizes contextual grounding as transforming heterogeneous inputs such as the grounding source, video prefix, and viewer request into an executable constraint model for interactive video continuation.
• The paper introduces VIS-Ground, a framework comprising structured context abstraction to recover grounded states and cross-context dependencies, generation constraints induction to derive candidate-specific constraints, and constrained video generation to enforce these constraints through planning, verification, revision, and rendering.
• The paper introduces VIS-Bench, a benchmark for narrative- and knowledge-grounded interactive video storytelling, and experiments across Omni-1.1-Flash, Veo 3.1, and MiniMax-H3 show VIS-Ground achieves the highest composite score, with an average absolute improvement of 10.3 points over the strongest per-backbone baselines.

Introduction

Video storytelling is useful for explaining complex ideas through language, imagery, motion, and sound, but viewers often want to redirect a video after seeing earlier frames, such as by asking for consequences or concrete examples. Prior systems either generate long-form video from a fixed story or script or allow prompt-driven interactive generation, yet they do not explicitly check whether a new viewer request remains compatible with the grounding source and the already rendered video. The authors formulate this as Video Interactive Storytelling with Contextual Grounding and propose VIS-Ground, which performs structured context abstraction, induces generation constraints, and uses an iterative plan-verify-revise-render process to keep continuations source-faithful, story-consistent, and request-aligned. They also introduce VIS-Bench, a benchmark for evaluating these properties across narrative- and knowledge-grounded settings.

Dataset

• Dataset composition and sources The authors build VIS-Bench from CoQA by sampling 250 source passages from MCTest, Gutenberg, RACE, CNN, and Wikipedia. The benchmark covers two grounding settings: narrative grounding, where a core narrative such as a story outline, script, or plot description guides important characters and events, and knowledge grounding, where an external source such as a scientific paper, Wikipedia article, or document must be faithfully communicated. Of the sampled sources, 132 are narrative-grounded and 118 are knowledge-grounded.

• Key details for each subset Each source passage is converted into a scene-based story with aligned source facts. The resulting stories are rendered in two visual styles: cinematic realism and cutout animation. The authors generate 1,004 interaction instructions at designated interruption points, producing 2,008 planned instruction-style instances. All instances are assigned to evaluation, so no training split is used.

• How the paper uses the data VIS-Bench is used strictly for evaluation. Each sample includes the source passage, video prefix, viewer instruction, and relevant reference assets. Target facts and continuation requirements are reserved for evaluation rather than included as input.

• Processing details Generated videos undergo quality checks and repair. Instruction alignment and information leakage are audited before packaging. The final packaged samples include the source passage, video prefix, viewer instruction, and relevant reference assets.

Method

The authors introduce VIS-Ground, a framework for video interactive storytelling that treats contextual grounding as a context compilation problem. The main idea is to compile heterogeneous inputs into a grounded dependency model and use that model to control subsequent planning, generation, and repair.

The overall pipeline is illustrated below:

VIS-Ground proceeds in three stages. First, the grounding source, rendered video prefix, and viewer request are converted into grounded facts and cross-context dependencies. Second, these dependencies are projected onto a candidate continuation, where only the relations made relevant by the proposed future become generation constraints. Third, the activated constraints guide planning, script repair, rendering, and output-level correction.

Compiling Context into Grounded Dependencies

The grounding source, video prefix, and viewer request provide different views of an evolving situation. The source may establish factual or narrative relations, the rendered prefix determines the currently visible state, and the viewer request specifies how the story should continue. Instead of summarizing each input independently, VIS-Ground extracts structured grounded facts and recovers dependencies that emerge from their interaction.

The framework first transforms each input into typed predicates with arguments, temporal scopes, and evidence links. These predicates are organized into three complementary states:

• Video state $V$V represents what has been visually established in the rendered prefix, including entities, attributes, locations, possession, actions, and state transitions.
• Story state $H$H represents the evolving narrative state, including completed events, active goals, unresolved relations, causal dependencies, and information already conveyed.
• Source state $S$S represents atomic claims, events, definitions, and relations supported by the grounding source, while preserving qualifications, conditions, and supporting evidence.

The aligned states are compiled into a grounded dependency model

$$\mathcal{M} = (\mathcal{V}, \mathcal{H}, S, \mathcal{A}),$$M=(V,H,S,A),

where $\mathcal{A}$A contains cross-context alignments. From this aligned model, the method derives a dependency set

$$R = \mathrm{Derive}(\mathcal{M}).$$R=Derive(M).

Each dependency expresses how one established fact constrains another event or state. These dependencies include temporal precedence, causal prerequisites, state-transition requirements, persistence relations, compatibility conditions, and source-supported dependencies. Many of these relations are not explicitly stated in any individual input. They emerge only after the video state, story state, and source state are jointly grounded.

Projecting Dependencies into Generation Constraints

VIS-Ground induces constraints from the derived dependency set $R$R. The method does not directly optimize final-video evaluation metrics, and those metrics are not used as predefined categories for constraint induction.

Let $\mathcal{P}^{(k)}$P(k) denote a candidate continuation script. For each dependency, the framework determines whether the candidate introduces an event, relation, or state transition for which that dependency becomes relevant. An activated dependency is represented as a generation constraint

$$c_j = (s_j, a_j, b_j, e_j),$$cj​=(sj​,aj​,bj​,ej​),

where $s_j$sj​ specifies the entity and temporal scope, $a_j$aj​ is an applicability condition, $b_j$bj​ is the relation that must hold, and $e_j$ej​ contains the grounded evidence supporting the dependency. The constraint is activated when

$$a_j(\mathcal{P}^{(k)}, \mathcal{M}) = 1,$$aj​(P(k),M)=1,

and once activated it requires

$$b_j(\mathcal{P}^{(k)}, \mathcal{M})$$bj​(P(k),M)

to hold within scope $s_j$sj​.

This formulation makes constraints candidate-conditioned. For example, a possession dependency need not constrain a continuation unless the proposed script introduces an action that requires a particular character to possess a corresponding object. Similarly, a source-defined prerequisite is activated only when the candidate attempts an event whose validity depends on that prerequisite.

Each activated constraint is evaluated and returns one of three outcomes: pass, fail, or unknown. A pass means the continuation is compatible with the dependency. A fail identifies a conflicting event, state transition, or missing prerequisite. An unknown indicates that the available evidence is insufficient to determine compatibility. Because every constraint retains links to its supporting predicates and evidence, failures can be traced back to the contextual dependency that caused them and used to generate targeted revision feedback.

Constrained Video Generation

The grounded dependency model remains active throughout generation. It is not used merely as additional conditioning for the video generator. Instead, it provides a common basis for proposing, checking, and repairing the continuation.

The planner first proposes a candidate script conditioned on the grounded context:

$$\mathcal{P}^{(0)} \sim p_\theta(\mathcal{P} \mid \mathcal{M}, u).$$P(0)∼pθ​(P∣M,u).

The dependencies activated by $\mathcal{P}^{(0)}$P(0) are then instantiated as constraints and evaluated. When a constraint fails, VIS-Ground identifies both the conflicting portion of the script and the grounded dependency that it violates. This produces targeted feedback for revision rather than regenerating the continuation from scratch.

The revised script is generated as

$$\mathcal{P}^{(k+1)} = \mathrm{Revise}_\theta(\mathcal{P}^{(k)}, \mathcal{M}, F^{(k)}),$$P(k+1)=Reviseθ​(P(k),M,F(k)),

where $F^{(k)}$F(k) contains the failed or unresolved dependencies and their supporting evidence. Because revision may introduce new events or state transitions, constraint activation is recomputed after each update. Planning and verification therefore form an iterative dependency-guided repair process in which the relevant obligations evolve together with the candidate continuation.

Once a candidate script $\mathcal{P}^*$P∗ satisfies the activated script-level constraints, it is passed to the video generator:

$$\nu_{\mathrm{cont}} \sim p_\psi(\nu_{\mathrm{cont}} \mid \mathcal{P}^*, \nu_{\mathrm{prefix}}, \mathcal{G}).$$νcont​∼pψ​(νcont​∣P∗,νprefix​,G).

Script-level validity, however, does not guarantee that the rendered video realizes the intended events or state transitions. The framework therefore extracts the realized relations from $\nu_{\mathrm{cont}}$νcont​ and projects the same grounded dependencies onto the generated output. If the video fails to realize an activated dependency, such as an omitted object transfer, an incorrectly changed persistent attribute, or a visually absent prerequisite event, the corresponding evidence is converted into output-level feedback. The planner then repairs the script or makes the missing dependency more explicit before rendering again.

In this way, VIS-Ground uses a single grounded dependency model across both reasoning and generation. Context is first compiled into dependencies, dependencies are instantiated as constraints only when a proposed future makes them relevant, and violations are used to repair either the planned continuation or its rendered realization. The process terminates when the generated continuation satisfies the applicable grounded dependencies or the computation budget is exhausted.

Experiment

The paper introduces VIS-Bench, a benchmark for video interactive storytelling with narrative and knowledge grounding, and evaluates generated continuations on semantic and perceptual dimensions using automated judges across three video backbones. Results show that the proposed VIS-Ground approach improves contextual grounding over direct generation and agent-based baselines in both grounding settings and across backbones, while visual quality remains largely bounded by the underlying generator. Ablations indicate that structured state representations and instance-specific planning play complementary roles, and failure analysis shows the method sometimes prioritizes consistency with prior context over literal instruction following.

VIS-Ground achieves the highest composite score with every tested video-generation backbone, with the clearest gains concentrated in grounding-related metrics such as source faithfulness, story continuity, interaction fulfillment, and joint constraint satisfaction. These advantages appear in both narrative and knowledge grounding settings and average roughly ten percentage points over the strongest competing result. The method also shows much lower cross-backbone composite variation than Direct Generation, while not being uniformly best on every individual visual or interaction metric. Across grounding-related metrics, VIS-Ground improves over the strongest competing result by roughly 10 percentage points on average across backbones and settings. The composite score of Direct Generation spans more than 30 points across the three backbones, while VIS-Ground spans below 6 points, an approximate 82% reduction in cross-backbone variation. VIS-Ground is not best on every individual metric; some interaction fulfillment, visual consistency, and video quality comparisons favor other methods. Ablations show that replacing structured context with a free-form summary or using one-pass planning lowers the mean of narrative source faithfulness, story continuity, and interaction fulfillment.

VIS-Ground was evaluated across multiple video-generation backbones in both narrative and knowledge grounding settings, where it consistently achieved the highest composite score with the largest gains on grounding-related measures such as source faithfulness, story continuity, interaction fulfillment, and joint constraint satisfaction. It also reduced cross-backbone composite variation by roughly 82 percent compared to Direct Generation, indicating much more stable behavior across backbones, though it was not uniformly best on every individual visual or interaction metric. Ablations further show that replacing structured context with a free-form summary or using one-pass planning weakens narrative source faithfulness, story continuity, and interaction fulfillment.