One-sentence Summary

Researchers from Nanjing University, Nankai University, and University of Waterloo introduce OmniAssistBench, a benchmark for assistant-style interaction in Omni-LLMs that uses predefined priors from source videos and reverse-engineered Internet videos to simulate continuous interactions, with evaluations showing that Gemini-3-Pro reaches 66.4 out of 100, Qwen3-Omni-Instruct reaches 51.2, and current models reveal weaknesses in visual prompts, historical context, and delayed responses.

Key Contributions

• OmniAssistBench is a comprehensive benchmark for evaluating omni-modal large language models as interactive real-time video assistants.
• An annotation pipeline resolves interaction path divergence by extracting predefined priors from source videos. These videos are reverse-engineered into multi-turn clips to simulate continuous guided interactions.
• Experiments on OmniAssistBench show that Gemini-3-Pro achieves 66.4 out of 100. Analysis reveals failures in gesture following, long-context memory, delayed response timing, and cross-turn information retention.

Introduction

The authors focus on the shift from passive video understanding to real-time interactive Omni-LLM assistants, where a model’s guidance changes the user’s next actions and creates diverging interaction paths. Traditional benchmarks use static videos with fixed question-answer pairs, so they cannot adapt video inputs to unpredictable model responses or enumerate every possible interaction trajectory. To address this, the authors introduce OmniAssistBench, built by reverse-engineering existing Internet videos: they derive an explicit prior path from each source video, fix that path, and segment the video into multi-turn clips that simulate assistant-user interaction. The benchmark combines a basic perception tier with an advanced goal-oriented tier plus filmed real-world cases, and the evaluation exposes limitations in visual prompt following, long-term memory, delayed response, and cross-turn context retention.

Dataset

The authors construct OmniAssisBench, a fully manually annotated benchmark for evaluating interactive video assistants. All labels come from human experts, with more than 1,000 expert-hours invested.

• Data sources and collection

  • Raw footage is primarily sourced from YouTube, supplemented by curated clips from action recognition, instructional video, and emotion analysis datasets.
  • A small number of home-filmed samples are included for scenarios that are hard to find online.
  • Videos are selected based on detailed plots (e.g., a social scene with at least three people) rather than broad keywords, and the collection aims to uniformly cover common daily-life scenarios.

• Dataset composition

  • Each sample is an open-ended question-answer pair embedded in a video.
  • The ground truth consists of a full answer paragraph and 1–3 key point phrases that must appear in a correct response.
  • Questions are designed for two task tiers:
    • Basic Interactive Understanding: questions about objects or events directly visible in the video.
    • Advanced Interactive Understanding: questions driven by user goals and prior knowledge summarized from the video; target objects are not explicitly mentioned.
  • Three additional real-world scenario test cases (Meeting Simulation, Blind Assistance, Handicraft Process Tracking) combine multiple abilities in longer, scripted one-shot videos.

• Processing and format

  • To simulate online streaming, user prompts are converted to speech via TTS and temporally embedded at the very end of the input video. Visual prompts (gestures, handwriting) are inserted as picture-in-picture clips, and user goals appear as on-screen subtitles.
  • For non-temporal tasks, videos are manually trimmed to 30–180 seconds while preserving all instructional cues; all videos are standardized to approximately 1080p resolution.
  • During quality refinement, state-of-the-art MLLMs (e.g., Gemini-3-Pro) check annotation completeness. If a model produces a logically correct but differently phrased answer, it is manually verified and added as an alternative reference.

• Usage in the paper

  • OmniAssisBench is used exclusively as an evaluation benchmark. There is no training split.
  • Models are assessed on their ability to answer open-ended questions in a multi-turn, streaming-like setting.
  • Evaluation relies on a dual-factor metric: semantic similarity to the ground truth sentence and inclusion of all key points.

Method

The authors leverage a comprehensive four-phase pipeline to construct the dataset, encompassing scenario design and video collecting, QA design, video editing and user prompt embedding, and quality refinement.

As shown in the figure below:

During the scenario design and video collecting phase, human experts collaborate with LLMs to brainstorm key scenarios that challenge specific model capabilities. Raw footage is sourced primarily from YouTube and curated datasets, supplemented by home-filmed samples for plots that are difficult to locate online. The authors ensure the collected videos uniformly cover common daily-life scenarios.

Refer to the framework diagram regarding the topic distribution:

In the QA design phase, experts formulate questions based on source videos and task types. For Basic Interactive Understanding tasks, questions target specific objects or events. Conversely, Advanced Interactive Understanding tasks rely on user goals and summarized prior knowledge to establish a fixed interaction path. To address open-ended evaluation challenges, each sample is annotated with a Ground Truth Sentence and 1 to 3 Key Points. The evaluation metric assesses both semantic similarity to the ground truth and the inclusion of all Key Points, preventing overly rigid standards.

During video editing and user prompt embedding, questions are converted to speech via TTS and temporally embedded. Visual prompts like gestures and handwriting are inserted picture-in-picture, while user goals appear as on-screen subtitles. Videos for non-temporal tasks are trimmed to 30 to 180 seconds to accommodate context window constraints, and all videos are standardized to 1080p resolution. In the final quality refinement phase, state-of-the-art MLLMs assess annotation completeness, and logically correct but phrasing-divergent responses are manually verified and added as alternative references.

Guided by the classification of Basic and Advanced Interactive Understanding, the authors distill key abilities to formulate 7 major tasks with 16 sub-tasks.

Refer to the framework diagram for the task construction overview:

Basic Interactive Understanding focuses on fundamental perception tasks such as Social Perception (recognizing identities and emotions), Temporal Perception (recalling nonsalient details), Referential Perception (distinguishing referred objects), and Non-audio Prompt Following (interpreting gestures or text). Advanced Interactive Understanding evaluates higher-level capabilities like Context-Aware Response (providing actionable advice grounded in the specific environment), Proactive Response (determining when to answer), and Process Tracking (assisting with long-horizon tasks).

As shown in the figure below:

To address complex scenarios where large-scale data construction is impractical, the authors capture three real-world cases designed to evaluate multiple abilities within a single story.

Refer to the framework diagram for examples of these real-world cases:

These cases include Meeting Simulation, which tests long-term memory and speaker identification over a continuous video stream; Blind Assistance, which evaluates environment description and navigation for visually impaired users; and Handicraft Process Tracking, which combines multi-party social interaction with process tracking.

Experiment

The experiments evaluate closed and open multimodal models on OmniAssisBench using concurrent video and audio inputs, a unified assistant prompt with proactive abstention, and GPT-5-based rubric scoring. Results show that models generally understand verbal instructions but struggle with visual perception, gesture interpretation, long-term context retention, response timing, and task tracking, with the largest gaps in open-source and real-world scenarios. Ablations confirm that removing either modality usually hurts performance, although visual-only input improves proactive responses by reducing audio distraction, while judge model agreement is high and teacher forcing or lower video resolution brings only limited gains.

The dataset includes seven tasks spanning single-turn perception and multi-turn interactive scenarios. Tasks with more interaction turns generally involve longer videos and more question-answer pairs, with single-turn tasks averaging under 80 seconds and multi-turn tasks exceeding 200 seconds on average. Single-turn perception tasks (Social, Temporal, Referential) each have one turn and average video lengths between 40 and 77 seconds. Multi-turn tasks like Proactive Response (4 turns) and Process Tracking (5 turns) are associated with substantially longer videos, averaging 209 and 249 seconds respectively. Process Tracking contributes the most QA pairs (246) among all tasks, reflecting its complex, multi-step nature. Non-audio Prompt Following is a two-turn task with 76 QA pairs from only 30 videos, indicating a higher density of interactions per video.

On basic interactive understanding tasks, the leading proprietary model Gemini-3-Pro reaches an overall average of 66.4 out of 100, while the best open-source model, Qwen3-Omni-Instruct, scores 51.2, revealing a clear performance gap. Current MLLMs generally interpret verbal prompts but struggle to deliver fully accurate and comprehensive responses. Among proprietary models, Gemini-2.5-Pro achieves the highest Social Perception average (69.6), edging out Gemini-3-Pro (66.2), though both models exhibit lower scores on the CE sub-task (59.0 and 62.8). Non-audio Prompt tasks expose stark imbalances: MiMo-V2-Omni scores only 27.2 on GPF while reaching 62.6 on OPF, highlighting uneven capabilities within this category. The top open-source model trails the best proprietary model by more than 15 points on the overall benchmark, indicating a substantial accessibility–performance trade-off in interactive understanding.

Gemini-3-Pro records the strongest overall average and leads most evaluated task categories, with Gemini-2.5-Pro as the nearest proprietary competitor. Performance varies by capability, as MiMo-V2-Omni is strongest on CR and Doubao-Seed-2.0-lite leads process tracking, but these models do not match the leading model's overall level. The reported top open-source model scores below all proprietary models in the comparison, indicating a clear proprietary edge. Gemini-3-Pro achieves the best overall average and the best results in most task-level metrics, though it is not uniformly ahead. The top open-source model reported in the paper scores below the lowest proprietary model included in the comparison, showing a sizable capability gap.

The benchmark comprises seven tasks spanning single-turn perception and multi-turn interactive scenarios, with longer videos and more question-answer pairs in multi-turn settings. Evaluation of leading multimodal models shows that proprietary systems like Gemini-3-Pro substantially outperform open-source alternatives, revealing a clear capability gap across all task categories. While some models exhibit strengths in specific capabilities, overall interactive understanding remains challenging, and performance is often uneven across sub-tasks.