Command Palette
Search for a command to run...
VibePrevisBench: Blender에서의 4D 장면 생성을 위한 에이전틱 코딩 평가
VibePrevisBench: Blender에서의 4D 장면 생성을 위한 에이전틱 코딩 평가
초록
제어 가능한 실세계 비디오 생성 모델은 렌더링된 그레이박스 프록시 비디오를 조건으로 삼을 수 있으며, 이는 레이아웃, 모션, 카메라를 고정한다. 코딩 에이전트는 Blender에서 4D 장면을 재구성하여 이러한 비디오를 구축할 수 있지만, 이러한 비디오와 그 이면의 길고 멀티샷인 장면을 모두 점검하는 벤치마크는 존재하지 않는다. 우리는 4D 장면 재구성을 위한 벤치마크인 VibePrevisBench를 소개한다. 에이전트는 브리프를 실행 가능한 4D 장면으로 변환하고 그 프록시 비디오를 렌더링한다. 242개의 태스크는 스크립트, 사진, 연속 장면, 참조 비디오를 포함한 네 가지 입력 유형을 포괄한다. 각 결과물은 두 가지 평가 차원에 따라 채점된다. 규칙 기반 Scene Check는 코드를 다시 실행하고 내보낸 4D 장면을 숨겨진 정답 사실과 아홉 가지 차원에서 비교한다. VLM 기반 Screen Check는 VLM 심사자에게 통과한 사실들이 프록시 비디오에서 인식 가능한지 묻는다. 우리는 11개 최첨단 코딩 에이전트의 평가 결과와 궤적을 분석하고 세 가지 핵심 관찰을 도출한다. 첫째, 멀티모달 이해가 에이전트들을 구분하며, 특히 공간과 카메라 모션 이해에서 두드러진다. 둘째, 모든 에이전트는 기본적인 카메라 작업과 스테이징을 이미 숙달한 것으로 보이며, 도전적인 요청에서 실패한다. 셋째, 에이전트는 자신의 렌더링을 점검하지만 오류를 수정하지 못한다. 따라서 에이전트는 공간과 카메라에 대한 더 나은 시각적 이해, 도전적인 요청에 대한 명시적 점검, 그리고 브리프 대비 렌더링의 항목별 검증이 필요하다.
One-sentence Summary
Researchers from Tianjin University, Kling AI, Beijing University of Posts and Telecommunications, et al. introduce VibePrevisBench, a 242-task benchmark that evaluates coding agents converting briefs into executable Blender 4D scenes and rendered proxy videos, scores them via rule-based Scene Check and VLM-based Screen Check, and finds that frontier agents need better spatial and camera understanding and error correction.
Key Contributions
- VibePrevisBench provides 242 tasks across scripts, photos, continuations, and reference videos, requiring coding agents to reconstruct an executable 4D scene and render its proxy video.
- Rule-based Scene Check re-runs the code and compares the exported 4D scene with hidden ground-truth facts in nine dimensions, while VLM-based Screen Check asks a VLM judge whether facts that pass can be recognized in the proxy video.
- Analysis of 11 frontier coding agents shows that multimodal understanding of space and camera motion separates agents, challenging staging and filming requests remain difficult, and agents inspect their own renders but fail to correct their errors.
Introduction
Controllable real-world video generation increasingly depends on proxy video rendered from a 4D scene, which fixes layout, people, actions, cameras, and cuts. Coding agents can build these scenes by writing Blender code, but errors in the proxy propagate into the final real-world video. Existing benchmarks only partially evaluate this task: they use narrow inputs, focus on static content, or rely on screen-only evaluation, without checking the underlying 4D scene against ground truth for long multi-shot scenes with people. The authors introduce VibePrevisBench, a benchmark where agents turn scripts, photos, continuations, or reference videos into executable 4D scenes and proxy videos, evaluated through rule-based scene checks and VLM-based screen checks.
Dataset
VibePrevisBench dataset
The authors introduce VibePrevisBench as a 242-task benchmark for 4D scene reconstruction. Each task is a tuple (b,m,E,o): an English brief b, optional media m, a list E of places and subjects with ids, and output requirements o specifying the frame count, frame rate, and resolution of the proxy video. The benchmark covers seven scene domains and four input kinds.
Subsets and sources
- Scripts: 94 tasks. The scene is specified in prose alone, without distances or angles. Script briefs are relatively long, around 800 words.
- Reference videos: 86 tasks. Clips come from more than 27,000 source videos. The authors keep clips with several moving people, camera motion, or cuts, leaving 1,800 clips. Of these, 388 pass VLM checks for live action, visible events, and distinct subjects, and human reviewers choose the final video tasks.
- Photo tasks: 39 tasks. A total of 38 tasks draw on scan datasets with measured 3D boxes and camera poses. One photo task is estimated by the reconstruction pipeline.
- Continuations: 23 tasks. Each task shows the opening frame of a clip and describes the next seconds.
Schema and hidden ground truth
Every task pairs a brief with hidden ground-truth facts that can be checked individually. A fact is one checkable statement about the scene, such as where subjects stand, how places connect, who moves where and when, or how cameras frame, move, and cut. The facts belong to nine dimensions: entities, layout, spaces, picture structure, routes, events, camera, cuts, and physical validity. Picture structure is checked only for photo, continuation, and video tasks, because it compares depth, normals, and segmentation with reference frames. Appearance is never checked.
Construction and processing
- Collection and curation: Two frozen taxonomies set quotas for what a scene holds and how a task is asked. Video and continuation clips are selected from a large source pool through movement, camera, and cut filters, followed by VLM checks and human review.
- Task construction: For script tasks, a model builds a scene from a card and writes the script. Two blind readers from the Claude and DeepSeek families reconstruct the events from the script alone; claims neither recovers are reworded or dropped. Media briefs assign each place and subject an id and description. Human reviewers check and correct every brief and its hidden facts.
- Ground truth and checks: Scripts take hidden facts from the built scene, and photo tasks from scans. For videos, most continuations, and one photo task, a pipeline estimates ground truth by recovering shots, subjects, cameras, depth, and bodies and lifting them to 3D. Facts the pipeline cannot verify are marked unscorable and never count. Candidate tasks that did not separate 11 agents were dropped based on item-rest correlation and spread, not on any single agent score. Tasks, media, hidden facts, and scorer are frozen with checksums.
Scale and statistics
The 242 tasks are distributed as 94 scripts, 86 reference videos, 39 photo tasks, and 23 continuations. Each of the seven domains holds 19 to 46 tasks, and each of the 33 places appears in 3 to 13 tasks. Briefs range from 153 to 1,200 words, with a median of 368 words. A task carries 20 to 2,175 checks, with a median of 408 and 99,000 checks in total. Most tasks are demanding: 88% have two or more shots, 80% have a moving camera, 79% have two or more people, 69% have hand-object interaction, and 61% involve two or more places.
How the benchmark is used
VibePrevisBench is used for evaluation rather than model training. An agent returns a Blender program p. The benchmark runs p in a fresh sandbox, exports the 4D scene S frame by frame, and renders the proxy video V. The hidden ground-truth facts form a rubric used to score the scene.
- Rule-based Scene Check: It binds subjects and places by tagged ids and matches untagged objects by class. Script facts are relational, while media facts are metric and are compared after similarity alignment. Scores are corrected for chance and missing entities, then averaged over the dimensions a task exercises.
- VLM-based Screen Check: A VLM judge, Gemini, watches the proxy video at 2 fps blind to the agent and object names, retells the clip, then receives the brief and checklists from the hidden facts. It rates items as no, partial, or yes across recognition, action, visibility, and story. Photo tasks use recognition only.
- Overall score: The final score is (Scene+Screen)/2.
No train/validation/test split is described, since the dataset is a fixed benchmark whose scoring is reproducible from the released checksums.
Method
The authors formulate a task τ=(b,m,E,o) comprising an English brief b, optional media m, a list of places and subjects E, and output requirements o. The agent responds with a Blender program p. To evaluate this, the program is executed to generate a 4D scene S=({xe(t)}e∈E,c(t),K) and a proxy video V.
Refer to the framework diagram for the evaluation pipeline.
The pipeline ensures that the delivered program runs independently in a fresh sandbox, exporting the scene frame by frame and rendering the proxy video. This prevents agents from submitting pre-rendered videos that do not match the underlying code. The evaluation consists of two main checks. The Rule-based Scene Check verifies if the 4D scene is structurally correct. It binds subjects and places by tagged IDs and matches untagged objects by class. The score is calculated in three steps: computing a raw score Pb for a block of facts, correcting for chance and missing entities using Qb=clip((Pb−Rb)/(1−Rb))κb, where Rb is the chance rate and κb is the share of existing entities, and combining blocks into dimension scores. The VLM-based Screen Check, utilizing a VLM judge, assesses whether the correct scene elements are visible. The judge watches the proxy video and rates items across dimensions like recognition, action, visibility, and story. The final score is the average of the Scene and Screen scores.
The authors construct the benchmark in three stages to ensure high quality and reproducibility.
As shown in the figure below:
Stage 1 involves collection and curation from scan datasets and self-collected videos, filtered by expert models and VLMs into specific domains and places. Stage 2 focuses on task construction, where scripts are written spec-first and validated by blind readers from different model families, while media briefs are screened by humans. Stage 3 establishes ground truth and checks. For scripts, facts come from the built scene; for videos, a pipeline recovers shots, subjects, and cameras to lift them to 3D. Tasks that fail to separate agent performance based on item-rest correlation and spread are dropped, ensuring a robust benchmark frozen with checksums.
Experiment
The study evaluates 11 frontier coding agents in a Blender sandbox with a four-hour budget, scoring their ability to generate, render, and revise scenes across different input kinds. GPT-6 Astra leads overall at 77.7, but no agent approaches solving the benchmark, and cost does not track score, with cheaper models reaching near-leader performance. Multimodal understanding of space and camera motion separates agents more than coding ability, while all models handle common staging requests but turn challenging requests into typical choices. Agents frequently inspect their own renders yet cannot reliably detect errors, indicating that explicit item-by-item verification against the brief is needed rather than more image viewing.
The comparison shows that existing benchmarks are partial across input, staging, filming, and evaluation dimensions. Many support rendered-frame scoring but do not require staged people, timed actions, multi-shot filming, or checks of the reconstructed 4D scene against ground truth. Only VibePrevisBench satisfies all listed criteria. Several related benchmarks score rendered frames on screen, but most do not verify the underlying 4D scene against ground truth. Only VibePrevisBench has full coverage across the listed dimensions, while related benchmarks cover staging and filming only partially.
On the 242-task benchmark, GPT-6 Astra leads overall, narrowly ahead of GPT-6.1 Sol, Gemini 4 Argon, and Opus 5.5. These top four models cluster closely and outperform the next agents by at least 3.9 overall points, but no agent approaches a solved benchmark. Vision-heavy inputs widen model differences, while cost and rank are not tightly connected. GPT-6 Astra attains the highest overall score, followed by GPT-6.1 Sol, Gemini 4 Argon, and Opus 5.5. The top four agents lead the next best group by at least 3.9 overall points. GPT-6 Astra leads rule-based scene check on script, continuation, and video inputs, while Opus 5.5 leads on photo inputs. The best-to-worst rule-based scene check gap increases from 12.0 points on text-only scripts to 19.9 points on photos, with space and camera errors accounting for much of the photo gap. No agent solves the benchmark: only 35 of 242 tasks have any agent achieving a rule-based scene check score of 90. Cost and score diverge; some much cheaper agents reach close to the leader's overall score, and a high-cost model can rank below lower-cost competitors. Agents inspect many rendered frames but do not reliably correct errors, and more inspection is not associated with substantially better scores.
The evaluation comparison indicates that existing benchmarks are partial across input, staging, filming, and evaluation dimensions, while only VibePrevisBench covers all criteria and verifies the reconstructed 4D scene against ground truth. On the 242-task benchmark, GPT-6 Astra leads narrowly over GPT-6.1 Sol, Gemini 4 Argon, and Opus 5.5, but no agent approaches a solved benchmark, and vision-heavy inputs widen model differences. Cost and rank are not tightly connected, and agents inspect many rendered frames without reliably correcting errors.