One-sentence Summary

NJU, Kling AI, TJU, et al. introduce CodeCap-Bench, a benchmark comprising 323 videos across five categories of structured 2D content with fine-grained annotations and a multilevel evaluation framework, and show that executable code captions outperform natural language captions in reasoning accuracy and reconstruction fidelity, while proprietary multimodal large language models still struggle to capture complex temporal dynamics and open-source models lag substantially behind.

Key Contributions

• The paper proposes executable code as a complementary video captioning representation, with programs specifying graphical primitives, layout rules, and functions of time, and shows that code captions achieve superior reasoning accuracy and reconstruction fidelity over natural language captions.
• The work introduces CodeCap-Bench, a benchmark comprising 323 videos across five categories of structured 2D content, fine-grained focal-point and object-level spatio-temporal annotations, and a multi-level evaluation framework covering global visual similarity, object-level spatio-temporal alignment, and fine-grained visual information retention.
• Systematic evaluation of multimodal large language models on CodeCap-Bench shows that leading proprietary models demonstrate promising code captioning capabilities but still struggle to faithfully capture complex temporal dynamics, while open-source models lag substantially behind; agentic workflows improve rendering success rates but do not consistently enhance caption fidelity.

Introduction

The authors explore executable code as a complementary video captioning format for structured 2D videos, such as interface recordings, 2D games, visualizations, and motion graphics. Code captions can specify graphical primitives, layout rules, and temporal logic, making spatial structure, motion trajectories, and state changes more explicit than natural language descriptions. Prior natural-language captioning methods often require lengthy positional, temporal, and relational constraints, and no dedicated benchmark existed for evaluating how faithfully multimodal large language models generate code captions. To address this, the authors introduce CodeCap-Bench, a 323-video benchmark with fine-grained focal-point and spatio-temporal annotations, and evaluate generated code by rendering it into video to measure visual similarity, object-level spatio-temporal IoU, and fine-grained visual information retention.

Dataset

• The authors construct CodeCap-Bench through a three-stage pipeline: multi-source video collection, content filtering, and fine-grained annotation with human review.
• The dataset focuses on structured 2D videos where fine-grained visual details, spatial layouts, and temporal dynamics are central.
• Candidate videos are collected from public platforms including Bilibili, YouTube, Behance, and Vimeo.
• The benchmark covers five categories:
  • Desktop and application interfaces
  • 2D games
  • Visual design and media animation
  • Data and scientific visualization
  • Icons and abstract motion graphics
• Content filtering retains videos dominated by 2D layouts, shapes, text, and explicit object trajectories. Videos dominated by complex 3D viewpoints, detailed textures, or dense pixel-level dynamics are excluded.
• CodeCap-Bench contains 323 videos across the five categories.
• Video durations average 19.0 seconds, with a median of 17.0 seconds.
• Each video has an average of 16.8 annotated objects and 8.6 static and dynamic focal points.
• Annotated focal-point intervals have a mean duration of 3.3 seconds and a median of 1.8 seconds.
• Two annotation types are provided:
  • Focal-point annotations identify key static features or dynamic processes with descriptions and timestamps. Static focal points capture appearance, composition, spatial relationships, and text presentation. Dynamic focal points capture state changes, motion processes, rhythm, and temporal coordination.
  • Object-level spatio-temporal annotations record visible intervals and per-frame bounding boxes of key objects.
• Object-level spatial annotations are obtained through a video segmentation model guided by bounding-box prompts and positive/negative point prompts on representative frames.
• All annotations undergo human review and correction. Reviewers check focal-point representativeness, description accuracy, temporal localization, object identity consistency, visible intervals, and spatial boundaries.
• The provided sections describe CodeCap-Bench as a benchmark for evaluating how faithfully models express visual information and spatio-temporal relationships through executable code. They do not specify training splits or mixture ratios.

Method

The authors formulate video captioning with code as the task of expressing visual content and its temporal evolution through executable programs. Given a reference video $V$V of duration $T$T, the goal is to generate a code caption $C$C that captures its visual elements, spatial layouts, and temporal dynamics. Unlike natural language captions, code captions explicitly specify how visual elements are constructed, arranged, and updated over time using graphical primitives, layout rules, and temporal logic. Executing the code through a rendering engine $\mathcal{R}$R produces a video $\hat{V} = \mathcal{R}(C)$V^=R(C), whose visual states are evaluated against the reference video $V$V in terms of visual appearance, spatial structure, and temporal dynamics.

To support this task, the authors construct the CodeCap-Bench dataset through a structured three-stage pipeline comprising multi-source video collection, content filtering, and fine-grained annotation with human review.

As shown in the figure below:

In the data collection stage, candidate videos are gathered from public platforms such as Bilibili, YouTube, Behance, and Vimeo. The collection spans five categories: desktop and application interfaces, 2D games, visual design and media animation, data and scientific visualization, and icons and abstract motion graphics. These categories encompass diverse visual content and dynamic patterns, including window operations, page scrolling, object motion, dynamic charts, geometric construction, and shape transformations.

The content filtering stage refines the candidate pool based on scene geometry, appearance complexity, and motion structure. The authors retain videos dominated by 2D layouts, shapes, text, and explicit object trajectories, while excluding those dominated by complex 3D viewpoints, detailed textures, or dense pixel-level dynamics. This selection ensures the benchmark remains focused on visual content that is amenable to programmatic representation.

Finally, the annotation and human review stage provides two types of annotations: focal-point annotations and object-level spatio-temporal annotations. Focal-point annotations identify key static features or dynamic processes, complete with descriptions and corresponding timestamps. Static focal points capture appearance, composition, spatial relationships, and text presentation, whereas dynamic focal points capture state changes, motion processes, rhythm, and temporal coordination. Object-level spatio-temporal annotations record the visible intervals and per-frame bounding boxes of key objects. The authors obtain spatial annotations using a video segmentation model guided by bounding-box prompts alongside positive and negative point prompts on representative frames. All annotations undergo rigorous human review and correction to ensure focal points are representative and accurately localized, and that object-level annotations maintain consistent identities, correct visible intervals, and accurate spatial boundaries.

Experiment

The experiments evaluate code caption fidelity using visual similarity, object-level spatio-temporal alignment, and fine-grained visual information retention across seven multimodal models in agentic and non-agentic settings. They find that successful video production does not guarantee faithful preservation, dynamic information is consistently less preserved than static information, and gains from agentic workflows, thinking effort, and tool use are model dependent. The evaluation metric aligns strongly with human judgments, and code captions retain more useful visual and spatio-temporal information than natural language captions for reconstruction and question answering, with tool-assisted feedback helping correct temporal errors.

At medium thinking effort, GPT-6-astra shows the strongest fine-grained visual information retention in both agentic and non-agentic settings, with its Codex configuration leading on overall retention and object-level spatio-temporal IoU among agentic runs. High video production success rates do not guarantee visual fidelity, since some models with near-perfect success retain much less visual information. Dynamic visual content is consistently less faithfully preserved than static content, and agentic generation improves production success but not always retention. GPT-6-astra achieves the highest overall fine-grained visual information retention, and its Codex configuration leads in object-level spatio-temporal IoU among agentic settings. A perfect video production success rate does not ensure faithful visual preservation; GPT-6-sol with Codex reaches full success but much lower overall retention than GPT-6-astra. Dynamic visual information is retained less faithfully than static information across models and generation configurations. Agentic generation consistently raises video production success rates, but its impact on visual fidelity varies by model and can reduce overall retention for some models.

Reported quality scores show that static metrics are consistently higher than dynamic and overall metrics for both evaluated models across the two judge models. Overall scores are higher when judged by GPT-6-sol than by Gemini-3.1-pro, and the GPT-6-sol model leads in static and overall scores while dynamic scores remain close. Static scores are higher than dynamic and overall scores for both evaluated models under both judge models. The two judge models agree closely on static scores but diverge on dynamic and overall scores, with Gemini-3.1-pro assigning lower values. GPT-6-sol attains higher static and overall scores than DeepSeek-v4.1-flash, while dynamic scores are nearly tied.

Code-rendered videos achieve higher question-answering accuracy than natural language captions for all three caption models. The improvements range from about 2.8 to 10.4 percentage points, and the best code-rendered result approaches the original video baseline. Code-rendered video accuracy is consistently higher than natural language caption accuracy across all caption models. Opus-5 shows the largest gain and reaches accuracy closest to the original videos. The overall improvement ranges from about 2.8 to 10.4 percentage points.

Code rendering consistently achieves higher reconstruction fidelity than caption-to-video generation across all three caption models and all reported metrics. The advantage is strongest for appearance and keyframe structural similarity, while motion pattern similarity improves by a smaller margin. These results suggest executable code captions preserve finer visual and spatio-temporal structure than natural language captions. Across every caption model, code rendering outperforms caption-to-video generation on appearance, motion pattern, and keyframe structural similarity. The gains are most pronounced for appearance and structural similarity, with motion pattern similarity showing smaller but consistent improvements. Opus-5 with code rendering achieves the highest appearance and structural similarity scores among the reported settings.

The experiments compare fine-grained visual retention, judge-scored quality, question-answering accuracy, and reconstruction fidelity across video generation configurations. Agentic generation improves production success but not always visual retention, GPT-6-astra preserves visual information best, and dynamic content is less faithfully retained than static content across models. Static quality scores exceed dynamic and overall scores, with judge agreement closer on static metrics and divergent on dynamic and overall scores. Code-rendered captions consistently outperform natural-language caption-to-video baselines in question-answering accuracy and reconstruction fidelity, with the largest gains in appearance and keyframe structure.