One-sentence Summary

Researchers from University of Science and Technology of China, FrameX.AI, and Sun Yat-sen University introduce SemComp-Bench, a benchmark for outcome-oriented semantic task completion in video generation that includes SemComp-Data across six domains and a vision-language model based protocol reporting OA and GR scores, and they demonstrate that current models still struggle with task-relevant semantic grounding.

Key Contributions

• The paper formulates Semantic Task Completion Video Generation as a new task that requires generated videos to achieve instructed outcomes while preserving task-relevant semantic relationships with a reference image.
• A scalable four-stage curation pipeline is introduced to construct SemComp-Data, converting raw videos into image-text-video instances with paired brief and detailed instructions while preserving task authenticity and fine-grained attribute alignment.
• The paper presents SemComp-Bench, a VLM-based evaluation protocol that measures Outcome Achievement and Generation Reliability via evidence-grounded binary judgments, and experiments on representative video generation models reveal substantial limitations in these capabilities.

Introduction

Recent video generation models achieve strong visual fidelity and temporal coherence, but it remains underexplored whether they can complete an instructed outcome while staying semantically grounded in a reference input, such as folding a given banknote into an origami turtle rather than replacing it with an unrelated turtle. Existing benchmarks mainly assess visual quality, temporal consistency, subject identity, and physical plausibility, and they do not directly test joint outcome achievement and high-level semantic grounding. The authors formulate this challenge as Semantic Task Completion Video Generation and contribute SemComp-Data, a curated image-text-video dataset built from full-context real-world videos to ensure task feasibility and fine-grained alignment, along with SemComp-Bench, a VLM-based benchmark that measures Outcome Achievement and Generation Reliability through evidence-grounded binary judgments.

Dataset

The authors construct SemComp-Data from full-context videos in Koala-36M. Each instance is a structured image-text-video triplet:

• Reference frame $r_i$ri​: defines the initial context.
• Instruction pair $\mathcal{T}_i = (i_i^{\text{brief}}, i_i^{\text{detailed}})$Ti​=(iibrief​,iidetailed​): two aligned instructions describing the same intended outcome at different specificity levels.
• Outcome-centric video clip $o_i$oi​: depicts successful completion, drawn from the same task instance as the reference frame.

The dataset construction uses a four-stage pipeline:

• Candidate Filtering:

  • Source pool: Koala-36M.
  • Title-based keyword filtering removes narration-dependent videos such as talk shows, interviews, and news.
  • Frames are sampled into a mosaic-based video abstract for VLM categorization.
  • A VLM assigns each video to one of six domains: Arts and Precision, Beauty and Fashion, Crafts and DIY, Food and Cooking, Gardening and Pets, Sports and Fitness.
  • Videos with insufficient visual evidence are labeled Uncertain and discarded.
  • Reference and outcome states are manually defined per task category.

• State Mining:

  • Uses category-specific state definitions to locate reference and outcome frames.
  • State Grounding is frame-level timestamp localization, not full outcome-segment localization.
  • A VLM identifies the demonstrated task and expected outcome, then locates timestamps for both states.
  • Quality Checking verifies the pair by asking the VLM to identify the outcome frame from the unlabeled pair.
  • Pairs are discarded if the VLM cannot unambiguously identify the outcome frame.

• Video Extension:

  • Uses the verified outcome timestamp as a temporal anchor.
  • Shot detection and same-scene merging follow the Panda-70M procedure.
  • The core segment containing the outcome frame is merged with visually consistent neighboring segments until the clip reaches at least 3 seconds.
  • Outcome-centric clips in SemComp-Data average approximately 4.03 seconds.

• Instruction Structuring:

  • Produces two instruction variants from the reference frame and outcome-centric clip.
  • Normalized Relation Description creates a brief instruction of no more than 30 words using the template: Verb + Main Subject in $r_i$ri​ + Preposition + Main State in $o_i$oi​.
  • Attribute Selection chooses one of four alignment types: Object Element, Person Identity, Object Appearance, Scene.
  • The VLM identifies which reference attributes should be preserved or discarded.
  • Instruction Composition expands the brief instruction into a detailed instruction by adding the alignment type, preserve/discard constraints, and fine-grained outcome characteristics such as background, lighting, colors, shapes, and component-level details.
  • The two variants enable controlled evaluation of instruction specificity without changing the underlying reference or outcome.

The paper describes SemComp-Data as suitable for evaluation of instruction specificity and potentially useful for future task-specific training. It does not appear to detail a training split or mixture ratio in this section. The distribution of alignment types, domains, and high-frequency brief-instruction words is summarized in the paper.

Method

The authors construct SemComp-Data as a collection of structured image-text-video triplets $x_i = (r_i, \mathcal{T}_i, o_i)$xi​=(ri​,Ti​,oi​), where $\mathcal{T}_i = (i_i^{\text{brief}}, i_i^{\text{detailed}})$Ti​=(iibrief​,iidetailed​) represents an aligned instruction pair describing the same intended outcome at two levels of specificity. The reference frame $r_i$ri​ defines the initial context, the paired instructions specify the desired outcome, and the outcome-centric video clip $o_i$oi​ depicts successful completion. Both $r_i$ri​ and $o_i$oi​ are drawn from the same task instance, ensuring genuine association of visual elements. To achieve this, the authors develop a four-stage curation pipeline to construct standardized evaluation instances from full-context videos.

As shown in the figure below:

In the first stage, Candidate Filtering, the authors utilize the Koala-36M dataset as the source pool. They apply Title-Based Keyword Filtering to exclude narration-dependent videos, retaining candidates that are visually self-contained. Uniformly sampled frames are arranged into a mosaic representation called a video abstract. An advanced VLM assigns each video to one of six domains and an associated task category, discarding those labeled as Uncertain.

The second stage, State Mining, mines reliable reference-outcome frame pairs through State Grounding and Quality Checking. State Grounding is formulated as frame-level timestamp localization. Given a full-context video and state definitions, the VLM identifies the demonstrated task and locates timestamps matching the reference and outcome states. The extracted frame pair is then verified through QA-based Quality Checking, where the VLM identifies the outcome frame from the unlabeled pair. Pairs where the prediction does not match the grounded outcome are discarded.

In the third stage, Video Extension, the authors take the full-context video and the verified outcome timestamp. Following shot detection and same-scene merging procedures, the video is partitioned into segments with consistent camera viewpoints. Using the verified timestamp as a temporal anchor, an outcome-centric clip $o_i$oi​ is extracted by selecting the core segment containing the outcome frame and merging visually consistent neighboring segments until a minimum duration of 3 seconds is reached.

The final stage, Instruction Structuring, produces two aligned instruction variants through Normalized Relation Description, Attribute Selection, and Instruction Composition. To capture the essential reference-outcome relation, Normalized Relation Description constrains the VLM to generate a brief instruction $i_i^{\text{brief}}$iibrief​ of no more than 30 words following a specific template. In parallel, Attribute Selection determines which reference attributes to preserve or discard by selecting an alignment type from Object Element, Person Identity, Object Appearance, and Scene. Instruction Composition expands the brief instruction into a detailed instruction $i_i^{\text{detailed}}$iidetailed​ by combining the selected alignment attributes with fine-grained visual characteristics of the completed outcome. The resulting detailed instruction jointly specifies the desired outcome and its reference-grounding requirements. The two variants describe the same task instance at different levels of specificity, enabling controlled evaluation of instruction specificity.

The alignment-type and domain distributions of the constructed data, along with high-frequency words in the instructions, are summarized in the figure below:

Experiment

These experiments evaluate generated videos with SemComp-Bench, which uses structured binary questions to score Outcome Achievement and Generation Reliability across sampled frames. The setup tests open and closed source image-conditioned and text-conditioned models on curated SemComp-Core instances, with repeated VLM judgments to measure task fidelity and visual reliability. Key findings show that outcome achievement remains a major challenge even for the strongest models, while generation reliability is generally higher but limited by within-scene spatiotemporal instability. Image conditioning consistently improves semantic grounding and entity consistency, and detailed instructions improve task outcomes but can reduce coherence, indicating a tradeoff between specification and generation difficulty.

Under detailed instructions, HunyuanVideo-1.5-720P-I2V leads overall outcome achievement, followed by Wan2.2-I2V-A14B, while no evaluated model reaches a joint pass rate above 40 percent. Models show complementary strengths: Seedance 2.0 performs strongly in outcome realization and semantic grounding but is weaker on grounded entity consistency and global visual continuity, whereas Wan2.2-TI2V-5B shows the reverse profile with higher consistency and continuity but lower task fidelity. These patterns indicate that robust outcome achievement depends on both task fidelity and outcome-video validity. HunyuanVideo-1.5-720P-I2V achieves the highest overall outcome achievement, followed by Wan2.2-I2V-A14B, with relatively balanced pass rates across the four criteria. Seedance 2.0 has strong outcome realization and semantic grounding but lower grounded entity consistency and global visual continuity, while Wan2.2-TI2V-5B leads on the latter two criteria but is weaker on outcome realization and semantic grounding. The best overall outcome achievement remains below 40 percent, and complementary model profiles suggest a trade-off between task fidelity and outcome-video validity.

Under detailed instructions, Seedance 2.0 achieves the highest generation reliability score, and Wan2.2-I2V-A14B is the strongest open-source model. The top models perform well on visual clarity and artifact-free rendering, while within-scene spatiotemporal coherence is the weakest reliability criterion across all models. Generation reliability and outcome achievement rankings do not fully align: Seedance 2.0 leads reliability but is substantially weaker on outcome achievement, whereas HunyuanVideo-1.5-720P-I2V leads outcome achievement despite lower reliability. Seedance 2.0 records the best overall generation reliability score, with Wan2.2-I2V-A14B leading among open-source models. Within-scene spatiotemporal coherence is the main bottleneck, showing the lowest pass rates for every model. Visual clarity and artifact-free rendering are relatively strong, especially for the top-ranked models. Reliability rankings diverge from outcome achievement rankings, while Wan2.2-I2V-A14B places second on both dimensions.

Under detailed instructions, image-to-video variants consistently achieve higher overall outcome scores than text-to-video counterparts across the reported model families, with gains concentrated in semantic grounding, grounded entity consistency, and global visual continuity rather than outcome realization. In text-to-video settings, detailed instructions produce higher overall scores than brief instructions through stronger outcome realization and semantic grounding, while brief instructions tend to improve entity consistency and continuity but yield very low joint pass rates. The highest reported joint pass rate occurs in the HunyuanVideo image-to-video detailed setting. Image-to-video conditioning under detailed instructions leads to consistently higher overall outcome achievement than text-to-video conditioning across all three reported model families. The image-to-video advantage mainly comes from better semantic grounding, grounded entity consistency, and global visual continuity, while outcome realization remains broadly comparable. In text-to-video generation, detailed instructions outperform brief instructions on overall scores, but brief instructions often score higher on grounded entity consistency and global visual continuity. HunyuanVideo with image conditioning and detailed instructions records the highest overall outcome score among reported settings.

These experiments evaluate image-to-video and text-to-video models under detailed and brief instructions across outcome achievement and generation reliability criteria. HunyuanVideo-1.5-720P-I2V ranks highest on outcome achievement and Wan2.2-I2V-A14B ranks second, but no model reaches a joint pass rate above 40 percent. Seedance 2.0 leads generation reliability yet is weaker on outcome achievement, while image-to-video conditioning consistently improves overall outcomes over text-to-video conditioning mainly through semantic grounding, grounded entity consistency, and global visual continuity; within-scene spatiotemporal coherence remains the main reliability bottleneck.