One-sentence Summary

Researchers from Alibaba's Qwen Business Unit, ShanghaiTech University, HKUST, and other institutions propose AVA-Encoder, an agentic self-evolution auto-encoder that transforms video into a Film Knowledge Graph and reconstructs it via dual-loop textual-gradient optimization, achieving a 20.7-percentage-point absolute gain (a 73.1% relative improvement) over the strongest external baseline and, in a controlled policy-only setting, yielding a pseudo-trained Agentic Video Encoder policy that outperforms a carefully human-tuned policy while using 74.3% fewer shot-level and 70.1% fewer keyframe-level system-prompt tokens.

Key Contributions

• The paper introduces the Agentic Video Auto-Encoder (AVA-Encoder), an agentic auto-encoding framework that converts a video into a Film Knowledge Graph and reconstructs it back into video. A gated dual-loop textual-gradient optimization process uses reconstruction residuals to jointly improve the Film KG and a hierarchical Agentic Video Encoder.
• The Film KG representation explicitly captures entities, events, assets, and multimodal relationships in structured, agent-operable form, enabling querying, manipulation, and linked editing while producing aligned intermediate records and rendered outputs for high-quality agentic video creation trajectories.
• Experiments report a 49.0% overall reconstruction score, a 20.7-percentage-point absolute gain and a 73.1% relative improvement over the strongest external baseline at 28.3%; in a controlled policy-only setting, the pseudo-trained Agentic Video Encoder policy reaches 45.8% versus 44.4% for an independently human-tuned policy while using 74.3% fewer shot-level and 70.1% fewer keyframe-level system-prompt tokens. The release includes the complete AVA-Encoder framework, a reliable agentic video reconstruction benchmark, and the first dataset of high-quality Film KG representations.

Introduction

Agentic video creation systems built on foundation models can already draft stories, design keyframes, and generate videos, but they still cannot reliably produce cinematic-grade films because their base models lack the planning ability to coordinate scripts, characters, shots, and audiovisual elements. A key obstacle is that human-directed cinematic videos contain rich filmmaking knowledge in tightly coupled multimodal form, while agents need structured, editable representations such as text, code, plans, and graphs. Prior video representations only partially solve this: low-level pixels and latents preserve visual detail but are hard for agents to query or edit; captions are readable but lose structure; and existing scene graphs or knowledge graphs are designed for retrieval and QA, so they retain sparse facts rather than enough detail to recreate a shot. The authors propose the Film KG representation and AVA-Encoder, a self-evolving agentic video auto-encoding framework that encodes a film into an agent-native, text-centered knowledge graph and reconstructs the film from it. Reconstruction quality becomes the faithfulness signal, with a multi-level encoder, a hierarchical graph structure, and dual-loop textual gradient optimization. The work also contributes a reconstruction-faithfulness benchmark and a released Film KG dataset with graph-based editing support.

Method

The authors formulate agent-native video representation learning as an agentic auto-encoding problem. The Agentic Video Auto-Encoder (AVA-Encoder) framework maps, compresses, and reconstructs a high-dimensional cinematic video $V$V through three foundational components: an Agentic Video Encoder, a Film Knowledge Graph (KG) representation space, and a fixed Video Decoder. The overall process is governed by the equation $V \xrightarrow{E(\cdot; P)} G \xrightarrow{\text{Dec}} \hat{V}$VE(⋅;P)​GDec​V^, where the encoder $E$E maps the video to a structured intermediate Film KG representation $G$G under an encoding policy $P$P, and the static decoder $\text{Dec}$Dec sequentially produces the reconstructed video $\hat{V}$V^.

The Agentic Video Encoder addresses the challenge of preserving fine-grained cinematic information through three-level specialized understanding and inter-level context injection. Given an input video, the encoder first partitions it into a temporally ordered sequence of cinematic shots and selects motion-stable keyframes. It then performs hierarchical agentic understanding via three stages: film-level understanding captures global narration and cross-shot relations while initializing a shared registry for characters, scenes, and objects; shot-level understanding captures temporal, audiovisual, and camera dynamics; and keyframe-level understanding extracts fine-grained visual composition. To prevent stage-isolated misjudgments, inter-level context injection propagates higher-level context to finer stages, ensuring information flows explicitly from film to shot to keyframe. Finally, the structured textual results and registry links are assembled into the Film KG representation $G$G.

The Film KG representation $G = (N_G, \mathcal{E}_G, \mathcal{A}_G)$G=(NG​,EG​,AG​) is a discrete, text-centered graph designed to preserve complex information dependencies. It organizes multimodal content into structured-text nodes and linked multimodal assets. The graph contains ten node types, including a narrative hierarchy of Story, Event, and Shot nodes, along with shot-specific state nodes (Character, Scene, Object, Style, Camera, and Audio) and keyframe nodes. Typed edges preserve production, temporal, and semantic dependencies, such as hierarchical containment, temporal transitions, and causal relationships. The asset layer $\mathcal{A}_G$AG​ stores or references generated images, audio, and video assets, making the representation directly operable by agents for reasoning and editing.

To achieve reliable self-evolution toward high-fidelity video representations, the authors develop a dual-loop textual-gradient evolution framework. This framework uses the fixed decoder to expose reconstruction failures, translates them into fine-grained textual gradients, and applies each gradient to the appropriate optimization target.

The optimization operates through two complementary loops. The outer loop performs Data-Agnostic Encoding Policy Pseudo-Training to learn reusable cross-video encoding behavior. It updates the shot and keyframe components of the encoding policy $P$P by processing a stream of source videos. For each video, the framework reconstructs selected shots, identifies failed atomic facts via frozen QA banks, and formulates policy-level textual gradients. A language rewriting agent proposes a candidate policy update, which is verified against both current and historical replay data. An Anti-Forgetting Gate ensures that the candidate improves current-video performance while bounding visual and historical degradation before acceptance.

The inner loop performs optional Data-Dependent KG Representation Refinement at test time to correct input-specific reconstruction residuals. Holding the complete policy fixed, this loop directly adapts the Film KG representation $G$G of the current video. It diagnoses failures using either direct ground-truth versus reconstructed keyframe differences or shot-reward checklist mismatches. These failures are converted into asset-level textual gradients to revise specific image or video generation payloads. An Anti-Degradation Gate verifies the candidate representation, ensuring that localized corrections do not introduce regressions in already-correct content. Both loops rely on a reconstruction residual signal that separates detailed optimization feedback from the independent final evaluation protocol.

Experiment

The experiments evaluate AVA-Encoder through four research questions covering reconstruction fidelity, the contributions of hierarchical understanding and two optimization stages, film knowledge graph operability, and downstream reuse. Using fixed foundation models, shared representation budgets, and a ground-truth-anchored benchmark, the method outperforms all baselines across reconstruction directions. Ablations show that hierarchical film-shot-keyframe encoding and both optimization loops provide distinct gains, while acceptance gates prevent target-dimension trade-offs and forgetting. Qualitative graph-editing studies confirm that local edits propagate consistently across linked shots without disturbing unrelated assets, and the same representation improves downstream agentic video generation systems when supplied as a single textual input.

AVA-Encoder leads all compared methods on reconstruction fidelity across video, keyframe, video back-captioning, and keyframe back-captioning. Its largest advantages are in direct video and keyframe comparisons, while back-captioning directions improve more modestly but still favor AVA-Encoder. All methods score lowest on video back-captioning. AVA-Encoder improves over the strongest baseline by 21.1 points on video and 34.2 points on keyframe comparisons. For back-captioning, AVA-Encoder leads the best baseline by 13.9 points on video back-captioning and 11.6 points on keyframe back-captioning.

The ablation shows that hierarchical video understanding substantially outperforms naive single-level encoding, and the two optimization loops contribute complementary gains: removing either loop lowers overall reconstruction, while removing both lowers it further. Acceptance gates also improve performance across all comparison directions, and direct video and keyframe evaluations remain consistently stronger than their back-captioning counterparts. Hierarchical encoding with the initial policy provides a large overall gain over the naive single-level configuration. Data-dependent KG refinement and data-agnostic policy pseudo-training each improve overall reconstruction, and combining both loops yields the best result. Acceptance gates improve every comparison direction by preventing non-target and historical degradation. Direct video and keyframe comparisons consistently outperform video back-captioning and keyframe back-captioning across configurations.

A single textual injection of the complete AVA-Encoder representation improves overall downstream story-video quality for every evaluated agentic video generation framework. Gains are concentrated in character, plot, and style dimensions, while camera and audiovisual dimensions show smaller or no movement. The improvement occurs without framework-specific adapters or prompt tuning. Both MovieAgent and FilmAgent improve in Overall quality when the AVA-Encoder representation is supplied once as text. Character and style scores rise substantially for both frameworks, while audiovisual scores remain the lowest and barely change.

The first experiment evaluates reconstruction fidelity across video, keyframe, and back-captioning comparisons, showing that AVA-Encoder consistently outperforms baselines with the strongest gains in direct video and keyframe reconstruction. An ablation study validates that hierarchical video understanding, two complementary optimization loops, and acceptance gates all contribute to performance, while direct comparisons remain stronger than back-captioning across configurations. A third experiment tests downstream story-video generation by injecting the AVA-Encoder representation as text into agentic frameworks, finding consistent overall quality improvements for MovieAgent and FilmAgent, especially in character, plot, and style, while camera and audiovisual aspects change little.