One-sentence Summary

Researchers from Video Rebirth, National University of Singapore, Nanyang Technological University, and other institutions analyze the agent harness as an executable runtime that coordinates planning, tools, state, feedback, and budgets for video generation across agentic video generation, agentic world modeling, and recursive self-improvement, and they identify open challenges in autonomous video production and evaluation.

Key Contributions

• Introduces an agent-harness framework for agentic video generation, covering script and shot planning, asset management, shot production, and editing, and clarifying which evaluation aspects must be separated.
• Examines agentic world modeling through the harness, including how observation, control, execution, and state are mediated; hybrid designs where a game engine or a program written by a coding agent holds world state while a video model renders it; and reciprocal testing and improvement between agents and world models.
• Analyzes recursive self-improvement for video model development by distinguishing revisions of current outputs, updates to harness state and generator/evaluator parameters, and changes to the procedure that produces later updates, then identifies channels through which improvements can re-enter development.

Introduction

Video generators now produce increasingly realistic, controllable, and temporally coherent clips, but a generator alone does not organize the decisions needed to complete video tasks: creative briefs must become coherent shots, references must persist across scenes, takes must be inspected, edits must preserve correct work, and interactive video must continue from observations. Existing surveys examine related pieces such as spatiotemporal consistency, post-training and alignment, evaluation, visual agents, world models, and agent harnesses, but they do not treat the harness around a video generator as the unit of analysis across production, interaction, and model development. Current systems also tend to automate fixed data collection and training loops rather than improving the loop itself, and no surveyed system yet revises the procedure that trains and validates the generator. The authors address this gap with a harness-centered framework that links agentic video generation, video world-model interaction, and development toward recursive self-improvement, while also identifying evaluation requirements and open problems for that path.

Dataset

The authors review how video generation training data is curated rather than introducing one new dataset. They organize the landscape into three stages: data acquisition and retrieval, annotation and synthesis, and data refinement.

Sources and composition

• WebVid: millions of web video-text pairs used as a general video generation source.
• HD-VG-130M: a larger collection of high-definition videos for text-to-video generation.
• VideoUFO: YouTube videos retrieved based on topics extracted from real user prompts, improving coverage of requested content.
• Phantom-Data: cross-pair training data with reference images of the same subject in different visual contexts, designed to reduce copy-paste artifacts in subject-to-video generation.
• VC-Agent: video retrieval data filtered through user feedback, including accepted samples and rejection reasons; the resulting data are used to fine-tune CogVideoX.
• Panda-70M: candidate captions generated by multiple cross-modality teachers and selected by a trained video-text retrieval model.
• ShareGPT4Video: detailed video captions produced by comparing consecutive keyframes to describe action and camera motion changes.
• Vidu S2-Avatar: temporally ordered captions aligned with individual events through multiple agents.
• PhysInOne: synthetic videos covering diverse physical phenomena to improve physical plausibility.
• Follow-Your-Instruction: synthetic video data built by MLLM agents that construct 3D scenes, verify layouts across views, and plan temporal evolution.
• EdenLive: training pairs synthesized from agent-built proxy videos.
• OpenVid-1M: video data improved through quality filtering and recaptioning.
• Koala-36M: videos selected by a Video Training Suitability Score that combines multiple quality indicators.
• SolarWM: 1.43 million clips from ten datasets unified into records containing observations, metric camera geometry, captions, quality metadata, selection decisions, and provenance.
• Ditto and JAVEdit: agent-driven filtering, correction, and pipeline repair for synthetic video data.

Processing and metadata

• The main processing steps are filtering, recaptioning, quality scoring, annotation, and synthetic generation.
• SolarWM separates source processing from mixture construction, so a new training mixture can be defined by changing filtering criteria or sampling weights without reprocessing the sources.
• The text does not specify a uniform cropping strategy; the emphasis is on caption quality, filtering, metadata unification, and provenance tracking.

How the data are used

• Curated data support training and fine-tuning of video generation models.
• VC-Agent data are used to fine-tune CogVideoX.
• PhysInOne, Follow-Your-Instruction, and EdenLive are used as synthetic supervision sources for improving physical plausibility or fine-tuning video models.
• The paper notes that provenance and the mixture of synthetic and real data matter for recursive training because repeated reuse of generated samples can reduce coverage, quality, or diversity.

Method

The authors propose a unified framework for agentic video generation and world modeling, centered on the distinction between the agent's decision process and the harness's execution runtime. An agent is defined as a goal-conditioned decision process where a controller policy $\pi_{\phi}$πϕ​ selects actions from observations. The harness $\mathcal{H}$H serves as the executable runtime that constructs controller context, exposes tool interfaces, dispatches operations, maintains state $s_t$st​, and enforces constraints. In each decision cycle, the controller proposes actions, the harness governs execution, and tools produce consequences, with feedback routing back to update the state.

In the context of agentic video generation, this framework orchestrates the transformation of a creative brief into an edited audiovisual work.

Refer to the framework diagram:

The workflow is organized into script and shot planning, asset management, shot production, and editing. Scripts define narrative requirements, while reusable assets guide the generation of candidate takes. The agent harness connects these activities, maintaining project state to coordinate changes across stages. Decisions propagate across artifacts; for example, modifying a character reference necessitates revisions in multiple shots, and the harness manages these dependencies to ensure consistency.

The framework extends to agentic world modeling, where video generation models serve as predictive cores for sustained interaction.

As shown in the figure below:

In this setting, one-shot conditioning evolves into sustained control. The harness observes the environment, decides on actions, translates controls, and incorporates returned feedback. The video world model predicts subsequent visual observations conditioned on actions and temporal context, allowing the agent to continually observe, act, and revise plans.

An alternative approach to world modeling involves explicit world state and generative rendering, which separates dynamics from the generator.

Refer to the framework diagram:

Here, a coding agent authors a world program that maintains state and applies interaction rules. An engine converts this state into proxy frames, which a video generator then renders into realistic imagery. This division of labor allows the harness to read and edit the world state directly from the program, rather than estimating it from frames, enabling precise control over geometry and motion while delegating appearance synthesis to the neural model.

To support these agentic systems, the authors outline a taxonomy for foundation model improvement.

As shown in the figure below:

This process encompasses data curation, including acquisition, annotation, and refinement, alongside model training strategies such as pre-training, supervised fine-tuning, reinforcement learning with reward modeling, and distillation. These mechanisms facilitate the development of more capable generators and evaluators, which in turn provide better supervision and feedback for the agent harness.

The overall agent design integrates these components into a cohesive architecture.

Refer to the framework diagram:

This design encompasses foundational capabilities like omnimodal perception, memory management, and long-horizon planning. It defines agent-environment interaction through model invocation, explicit code representations, and GUI control. Feedback-guided control mechanisms handle evaluation, failure diagnosis, and acceptance criteria. Furthermore, the architecture supports agent experience and skill reuse, allowing the system to adapt via in-context learning and deployable skills, as well as multi-agent organization for task delegation and collaborative review. Together, these aspects govern how a video agent pursues a goal through successive generation and revision decisions.

Experiment

These sections describe evaluation frameworks for agentic world modeling and recursive self-improvement in video generation. Agentic world modeling is assessed through active agent-driven testing of long horizon coherence, intervention sensitivity, and constraint consistency, while also treating world models as environments for testing agents and requiring separate measurement of world reliability and agent capability. Self-improvement evaluation combines video output assessment using benchmarks, model judges, human assessment, and agentic pipelines with agent harness evaluation across effectiveness, diagnosability, generalization, and continual adaptation. Across these settings, the work emphasizes keeping development feedback separate from held-out validation, controlling resource budgets and baselines, and confirming failures through targeted interventions rather than relying on visual plausibility alone.

The comparison examines related surveys and adjacent studies across scope, thematic coverage, and agent design. Existing studies consistently cover world models but vary or lack coverage in video generation, agentic generation, and agent-world interaction. The proposed survey is positioned as complementary by uniting video generation, world models, model development, and recursive self-improvement through a unified agent-harness framework. World-model coverage is consistently present across the compared studies. Agent-world interaction is not covered by any compared study, whereas it is central to the proposed framework. The proposed framework treats the agent harness around the generator as the unit of analysis, linking the video model as synthesizer, as interactive environment, and as object of development toward recursive self-improvement.

The notation defines a conditional video generator that produces a video sequence from conditions and stochastic variation, with conditions spanning text, images, keyframes, masks, depth, pose, trajectories, camera parameters, audio, or actions. It distinguishes sampling time, token position, and interaction time, and it treats a world model as a functional role rather than a separate sampler family. Conditions are not necessarily fixed by a user and may be constructed, decomposed, or revised by a controller during interaction. A conditional generator models video as a probability law over sequences given conditions and stochastic variation. Conditions can include many visual, audio, and control signals, and may be composed or revised by a controller. The notation separates sampling time, token position, and interaction time, with world modeling defined by its functional role. Interactive world models extend video continuation by receiving observations and action conditions and returning generated observations for subsequent steps.

The table defines the functional components of an agent harness, including observation, context construction, available tools, action execution, and the controller policy. These components are conceptual roles and need not be separate networks or software modules. The accompanying discussion clarifies that agency arises from an execution process, while using an LLM or VLM is an implementation choice rather than the definition of agency. The agent harness is decomposed into functional roles such as observation, context construction, tools, and action execution, which may be combined within a single module. The controller policy is distinct from generator parameters, separating the decision process from the underlying model. Agency is treated as a goal-conditioned execution process that can use learned, programmed, or hybrid policies, with no required reliance on an LLM or VLM.

The comparison shows that most listed video world models have no external agent harness in their runtime configuration, with only AdaWorld and Genie 3 listing external harness roles. Control interfaces vary from latent or explicit actions to combinations that include camera poses, text prompts, and, in the wider discussion, semantic or object interactions. The accompanying text describes harness evidence as thin, with most mechanisms demonstrated in single systems and few isolated quantitative gains attributable to the harness. Most listed systems are marked as having no external runtime harness; AdaWorld includes interaction support and Genie 3 includes interaction and evaluation support. Control interfaces expand from single latent or action controls to combinations such as action with camera pose or action with text, while the discussion notes that harness-specific gains are rarely isolated.

The evaluation compares related surveys, formalizes a conditional video generation and world model notation, decomposes the agent harness into functional roles, and surveys runtime harness support in existing video world models. Prior surveys consistently cover world models but largely omit agentic generation and agent-world interaction, positioning the proposed unified agent-harness framework as complementary. The notation clarifies that world modeling is a functional role, conditions may be constructed or revised by a controller, and interactive world models extend video continuation through observations and actions. The system review finds that most listed video world models have no external runtime harness, with only AdaWorld and Genie 3 listing harness roles, and harness-specific gains are rarely isolated.