HyperAIHyperAI

Command Palette

Search for a command to run...

ALIVE: ストリーミング動画モデルをインタラクティブにする

概要

ストリーミング動画LLMは世界を継続的に観察できるが、ユーザーの要求を個別に処理することが多い。本研究では、プロアクティブな監視、ストリーミングタスクの調整、フィードバックを求める対話を支援するインタラクティブなストリーミング動画モデルALIVEを提案する。2段階の学習のためにALIVE-SFT-288KとALIVE-RL-3Kを構築した。対照的教師ありファインチューニングにより、いつどのように応答するかを学習し、Behavior-Aligned Credit Assignment(BACA)を用いた強化学習では、軌跡レベルの利点と局所的な利点を組み合わせて、マルチターン対話における判断、応答、沈黙を洗練させる。また、フィードバック要求とその後のタスク完了を評価するFSI-Benchを導入する。ALIVEは、4つの公開ベンチマークとFSI-Benchにおいて比較モデルの中で最高の総合スコアを達成し、優れたストリーミング理解力とインタラクティブ能力を示した。

One-sentence Summary

Researchers from Tsinghua University, Peking University, ByteDance, et al. propose ALIVE, an interactive streaming video model that enables proactive monitoring, streaming task coordination, and feedback seeking, trained via contrastive supervised fine-tuning and reinforcement learning with Behavior-Aligned Credit Assignment (BACA) to refine multi-turn decisions, responses, and silence, achieving the highest overall scores on four public benchmarks and FSI-Bench.

Key Contributions

  • ALIVE is introduced as an interactive streaming video model supporting proactive monitoring, streaming task coordination, and feedback-seeking interaction within a continuous video stream.
  • Two training datasets, ALIVE-SFT-288K and ALIVE-RL-3K, support a two-stage approach in which contrastive supervised fine-tuning teaches when and how to respond, and reinforcement learning with Behavior-Aligned Credit Assignment (BACA) refines decisions, responses, and silence across multi-turn interactions.
  • FSI-Bench is presented to evaluate feedback seeking and subsequent task completion, and ALIVE achieves the highest overall scores among compared models on four public benchmarks and FSI-Bench.

Introduction

Advances in large language models are pushing conversational AI toward multimodal interactive assistants that continuously perceive video streams and interact with users. However, current streaming video LLMs remain largely passive: they often respond to individual requests in isolation and lack the ability to maintain or revise user tasks or proactively engage as events unfold. This gap limits their potential as foundations for interactive assistants. The authors formulate interactive streaming video understanding around proactive monitoring, streaming task coordination, and feedback-seeking interaction. They introduce ALIVE-SFT-288K and ALIVE-RL-3K for two-stage post-training, along with FSI-Bench for evaluating feedback-seeking interaction. Their ALIVE model combines contrastive supervised fine-tuning with reinforcement learning using Behavior-Aligned Credit Assignment, enabling an offline video LLM to preserve strong video understanding while learning to manage multiple tasks, modify or cancel tasks, and proactively seek user feedback during streaming.

Dataset

The authors construct two interaction datasets for streaming tasks: ALIVE-SFT-288K and ALIVE-RL-3K.

  • Dataset composition and sources

    • Built from public video segments and annotations from diverse sources, including TimeLens2, LLaVA-Video, TimeLens, Shot2Story, VideoGPT+, VideoRefer-700K, QVHighlights, Vript, WebVid, DiDeMo, Ref-YouTube-VOS, Nymeria, EPIC-KITCHENS, Charades-Ego, Ego4D, HoloAssist, Ego-Exo4D, DenseStep2M, COIN, HACS, Charades-STA, STAR, ActivityNet, and Oops.
    • Each example pairs video segments with timestamped user requests, feedback, assistant responses, or silence to form interaction trajectories.
  • Subsets and scale

    • ALIVE-SFT-288K: 288,436 interaction examples for supervised fine-tuning.
    • ALIVE-RL-3K: 3,523 interaction examples for reinforcement learning.
    • Both datasets span diverse scenarios and cover eight task types across proactive monitoring, streaming task coordination, and feedback-seeking interaction.
  • Task taxonomy

    • Proactive Monitoring: Proactive Timing, Proactive Understanding, Repeated Proactiveness.
    • Streaming Task Coordination: Task State Update, Concurrent Requests.
    • Feedback-Seeking Interaction: Ask-for-Monitor, Ask-for-Information, Ask-for-Execution.
  • Processing and filtering

    • Temporal evidence grounding and refinement: uses existing descriptions and approximate timestamps when available. Otherwise, an offline MLLM first detects relevant targets and coarsely localizes their occurrence. An offline MLLM then refines each candidate using video context before and after the candidate event, identifying the earliest sampled time at which the required visual condition is met.
    • Dialogue synthesis and temporal alignment: grounded targets and timestamps are used to synthesize user requests, assistant responses, task updates, and feedback exchanges. These are aligned with the evolving task state and visual evidence.
    • Quality verification: filters examples based on interaction-strategy validity and causal answerability. Answers must be supported by video observed up to the assigned timestamp and the preceding interaction history, without relying on future frames or later user feedback.
    • The construction focuses on temporal grounding rather than spatial cropping. No additional spatial crop metadata is described.
  • Usage in the model

    • ALIVE-SFT-288K is used for supervised fine-tuning.
    • ALIVE-RL-3K is used for reinforcement learning.
    • The provided section does not report additional training mixture ratios or split details.

Method

The authors construct the ALIVE-SFT-288K and ALIVE-RL-3K datasets to train interactive streaming video models. This process pairs public video segments with timestamped user requests, feedback, and assistant responses to form comprehensive interaction trajectories. The construction pipeline proceeds through temporal evidence grounding, dialogue synthesis and alignment, and rigorous quality verification.

For temporal evidence grounding and refinement, the authors leverage offline multimodal large language models to detect and coarsely localize relevant targets. These candidates are then refined using bidirectional temporal context to identify the earliest sampled time at which the required visual condition is met, such as entity recognizability or action completion. Following this, dialogue synthesis and temporal alignment generate interactions that pair user requests with appropriate assistant responses, ensuring each answer follows the necessary observations or user feedback. Finally, quality verification filters out examples that fail interaction-strategy validity or causal answerability checks, ensuring that each answer is supported strictly by the video observed up to its assigned timestamp and the preceding interaction history.

To establish the conditional interaction behaviors required for streaming, the authors employ Contrastive Supervised Fine-Tuning. At each decision time, the autoregressive model generates an output containing a control token that indicates whether to respond or remain silent. The authors construct two complementary families of controlled contrasts to supervise these dependencies. Response-gate contrasts hold the visual prefix fixed while varying interaction conditions, such as query relevance or task status, to teach the model when to respond versus remain silent. Action-strategy contrasts vary evidence availability or referential ambiguity to teach strategy selection, such as answering directly, requesting clarification, or asking for user execution. During interaction-aware training, they sample contiguous visual windows and balance response and silence supervision. The supervised fine-tuning objective is formulated as:

LSFT=−∑j∈Bmjwjlog⁡pθ(yj∣x<j)∑j∈Bmjwj\mathcal{L}_{\mathrm{SFT}} = - \frac{\sum_{j \in \mathcal{B}} m_{j} w_{j} \log p_{\theta}(y_{j} \mid x_{< j})}{\sum_{j \in \mathcal{B}} m_{j} w_{j}}LSFT​=−∑j∈B​mj​wj​∑j∈B​mj​wj​logpθ​(yj​∣x<j​)​

where mjm_{j}mj​ selects supervised tokens, wjw_{j}wj​ is the corresponding weight balancing response and silence tokens, and x<jx_{< j}x<j​ denotes the preceding multimodal context.

To further refine these behaviors on model-generated trajectories, the authors introduce Reinforcement Learning with Behavior-Aligned Credit Assignment. Starting from the supervised policy, they sample trajectories by replaying video observations and user turns chronologically. Each trajectory receives a reward that evaluates the required interaction strategy, aggregates task completion, and applies a redundancy penalty factor to downweight trajectories with excessive responses. To provide granular learning signals, they construct local behavioral units capturing strategy selection, task response, and critical silence. These units allow the computation of local advantages, which are combined with trajectory-level global advantages. For a specific model call, the combined advantage is calculated as:

Aic=AiE+αAi,u(i,c)BA_{ic} = A_{i}^{E} + \alpha A_{i, u(i, c)}^{B}Aic​=AiE​+αAi,u(i,c)B​

where AiEA_{i}^{E}AiE​ is the trajectory-level advantage, Ai,u(i,c)BA_{i, u(i, c)}^{B}Ai,u(i,c)B​ is the local advantage for the assigned behavioral unit, and α\alphaα weights the local signal. The policy is then optimized using these combined advantages to preserve supervision on overall interaction quality while providing distinct learning signals for specific behavioral requirements.

Experiment

The study evaluates ALIVE, initialized from Qwen3-VL-8B-Instruct and trained with a two-stage SFT and RL procedure, on public streaming video QA benchmarks and a newly introduced FSI-Bench for feedback-seeking interactions. Public benchmark results show ALIVE achieves state-of-the-art overall performance, particularly on tasks requiring interactive capabilities and adaptation to changing user states. On FSI-Bench, ALIVE outperforms proprietary and open-source baselines under both explicit and free-form decision protocols, though it remains below human performance, indicating substantial room for improvement. Ablations confirm that contrastive SFT and the BACA local advantage mechanism jointly improve streaming interaction capabilities.

Existing streaming video training datasets consistently provide proactive monitoring, but explicit support for streaming task coordination and feedback-seeking interaction is rare. ALIVE-SFT is the only listed dataset that covers all three interaction types, and the related ALIVE-RL resource extends this coverage to reinforcement learning. The datasets vary widely in scale and multi-turn design, and all listed entries are supervised fine-tuning resources. Every compared dataset includes proactive monitoring support, while only ALIVE-SFT explicitly supports streaming task coordination and feedback-seeking interaction. ALIVE-SFT is a multi-turn dataset with visual sources from both egocentric and non-egocentric settings, unlike several larger datasets that are not multi-turn. All listed datasets are supervised fine-tuning resources, and the largest listed dataset is more than twenty times the size of the smallest.

Across the reported public streaming video benchmarks, ALIVE achieves the strongest overall performance among the compared models. Baseline results vary widely, with AURA generally leading the listed baselines and earlier streaming models trailing substantially on real-time visual perception tasks. The largest gain for ALIVE is on IPIBench, where it surpasses the strongest baseline by a notable margin. ALIVE achieves state-of-the-art overall performance on all four public benchmarks compared with the evaluated models. Among baselines, AURA tends to be the strongest, while VideoLLM-Online and Flash-VStream score much lower on real-time visual perception tasks. ALIVE's largest improvement over the best baseline occurs on IPIBench, with an 18.7 point advantage over AURA.

On FSI-Bench, ALIVE achieves the highest overall accuracy under both the explicit and free-form protocols, surpassing the strongest proprietary baseline by 10.6 and 23.9 points respectively. Proprietary models are comparatively strong at direct answering but weak at asking for information and asking for execution. Despite ALIVE's lead, its free-form performance still falls well below the human reference. ALIVE's gains over the strongest proprietary baseline are larger under the free-form protocol than under the explicit protocol. Proprietary models perform relatively well on direct-answer tasks but struggle on feedback-seeking tasks, and open-source streaming models remain poor on those tasks except ALIVE.

Contrastive SFT improves every reported aggregate score relative to SFT without contrastive examples, with the largest gain on FSI-Bench under FDP. Adding trajectory-only RL further improves most scores but slightly lowers IPI performance compared with contrastive SFT alone. The full model with local behavioral advantages achieves the highest scores across all benchmarks and both FSI-Bench protocols. Contrastive SFT improves all reported aggregate metrics over non-contrastive SFT. Trajectory-only RL improves most evaluations but slightly reduces IPI performance relative to contrastive SFT. The full model with local behavioral advantages records the highest scores across all six evaluations and both FSI-Bench protocols.

The experiments evaluate streaming video dataset coverage, public benchmark performance, FSI-Bench interaction protocols, and training-stage ablations. ALIVE-SFT is the only dataset that combines proactive monitoring, streaming task coordination, and feedback-seeking support in a multi-turn setup, and ALIVE achieves the strongest results across public streaming benchmarks and FSI-Bench, with particularly large gains on feedback-seeking tasks. Ablations indicate that contrastive SFT consistently improves over non-contrastive SFT, trajectory-only RL helps in most evaluations, and the full model with local behavioral advantages performs best overall.


AIでAIを構築

アイデアからローンチまで — 無料のAIコーディング支援、すぐに使える環境、最高のGPU価格でAI開発を加速。

AI コーディング補助
すぐに使える GPU
最適な料金体系

HyperAI Newsletters

最新情報を購読する
北京時間 毎週月曜日の午前9時 に、その週の最新情報をメールでお届けします
メール配信サービスは MailChimp によって提供されています