One-sentence Summary

Researchers from USTC, Xiaohongshu Inc., and other institutions propose Video-DeepResearch, a multimodal deep research agent that extends to continuous video streams via a decoupled perception-exploration pipeline with stagewise tool unlocking, trained with supervised fine-tuning and Group Relative Policy Optimization (GRPO), and evaluated on VIDEODR-BENCH where Video-DeepResearch-35B-A3B achieves a new state-of-the-art $64.0\%$64.0% average accuracy, surpassing GPT-5 and Gemini 2.5 Pro.

Key Contributions

• The Video-DeepResearch framework employs a decoupled perception-exploration pipeline with stagewise tool unlocking to enforce exhaustive visual grounding before web retrieval, addressing modality bias and parametric knowledge leakage, and uses a two-stage SFT-GRPO training recipe for autonomous exploration.
• VIDEODR-BENCH is a human-AI collaborative benchmark of 200 multi-hop VQA instances where every question demands both dense spatiotemporal video reasoning and external knowledge retrieval.
• Video-DeepResearch-35B-A3B achieves state-of-the-art 64.0% average accuracy on VIDEODR-BENCH, surpassing Claude-4.5-Sonnet (59.0%), GPT-5 (52.5%), and Gemini 2.5 Pro (57.5%), while the 30B-A3B variant scores 59.3%, demonstrating competitive performance and training paradigm effectiveness at compact scale.

Introduction

The authors tackle the emerging challenge of Video-DeepResearch, where AI agents must conduct complex, multi-step investigations within continuous video streams to enable more comprehensive autonomous research in visually rich environments. Prior work has advanced from text-only to image-based deep research, but existing agents exhibit a systematic aversion to active visual exploration and are prone to exploiting memorized knowledge instead of genuinely reasoning over video evidence. To remedy this, the authors introduce VIDEO-DEEPRESEARCH, a unified framework featuring a scalable data engine with stage-wise tool unlocking, and they further contribute VIDEODR-BENCH, a rigorous benchmark requiring both visual search and external knowledge reasoning for every question.

Dataset

The authors construct two video-question-answering datasets to train and evaluate a model that answers questions by using external search tools.

Synthetic Training Set: 30 k VQA Pairs

• Sources Videos are collected from a broad mix of established video benchmarks and real-world streaming platforms, covering many domains.

• Filtering rules A two-stage pipeline cleans the pool. Rule-based: videos outside predefined duration limits are discarded. Agentic: a Qwen3.5-35B-A3B model rates content complexity and removes videos that are uninformative or overly simple. The filtered pool is split into a training subset and a test subset; the test subset later seeds the evaluation benchmark.

• Metadata construction For each video, an agent-driven process (using Qwen3.5-397B-A17B) first selects keyframes via CLIP inter-frame similarity. The same model localizes distinct visual entities with bounding boxes. These crops are used for visual search, and a secondary model (Qwen3.5-35B-A3B) verifies semantic alignment between the crop and the search results. The final metadata tuple is: ⟨keyframe, bounding boxes, entity name, search summary⟩.

• QA generation Questions are synthesized from the curated metadata following two patterns: Single-entity: fact-based questions about one entity. Multi-entity: compositional questions requiring reasoning across multiple entities. Superficial visual attribute questions (e.g., “what color is the car?”) are penalized.

• Memory-leakage filtering Each generated QA pair is stress-tested for parametric knowledge. The model attempts to answer the question four times without using any tools; if it succeeds in any attempt, the instance is permanently discarded. This ensures that the remaining 30 k pairs require external search for a correct answer.

• Usage This set is used to construct training trajectories, likely for supervised fine-tuning or reinforcement learning of the tool-using agent.

VIDEODR-BENCH: Evaluation Benchmark

• Origin A human-curated subset of the filtered video pool described above, carefully sampled and annotated to guarantee answerability and high data fidelity.

• Human-in-the-loop annotation Annotators watch videos and pause at critical timestamps. They use a Crop_Search tool to query salient visual entities, strictly verifying that the retrieved external evidence matches the original frame. From these verified results, they write several seed VQA pairs per video.

• Multi-agent synthesis for complex multihop questions Seeds are fed into a multi-agent pipeline:

  1. A Drafting Agent brainstorms semantic expansions (e.g., from “LeBron James” to related keywords about his team, spouse, or awards) and queries the web.
  2. A QA Generation Agent uses the retrieved contexts to formulate novel multihop questions.
  3. Leakage filter: any question that the model can answer without tool access is discarded.
  4. Human annotators verify answerability based on retrieved evidence.
  5. A Ranking Agent scores the validated candidates and keeps only the highest-rated question. This top-ranked question can itself become a new seed, enabling an iterative loop that escalates complexity to higher-hop reasoning tasks.

• Characteristics The benchmark contains complex, multi-hop questions that explicitly depend on external web and visual search, with all instances verified by humans for answerability and minimal parametric leakage.

• Usage VIDEODR-BENCH serves as the primary evaluation set to measure the model’s ability to perform open-ended video deep-research with tool augmentation.

Method

The authors formulate Video DeepResearch (Video-DR) as a sequential grounding pipeline that conceptualizes video as a temporal composition of key entity trajectories. To operationalize this workflow and bridge the gap from image-centric research, the agent is equipped with fundamental tools to isolate informative moments and construct precise visual queries. The overall framework of the Video-DeepResearch pipeline is illustrated below:

Formally, given a complex research query $Q$Q and a visual input $V = \{v_1, v_2, \ldots, v_T\}$V={v1​,v2​,…,vT​}, the goal is to synthesize a comprehensive response $R$R. This is modeled as a sequential decision-making process where the agent generates an action $a_i \in \mathcal{A}$ai​∈A based on the historical trajectory $\mathcal{H}_i$Hi​:

$$a_i \sim \pi_\theta(a \mid \mathcal{H}_i)$$ai​∼πθ​(a∣Hi​)

where $\pi_\theta$πθ​ denotes the multi-modal policy. The action space $\mathcal{A}$A encompasses the defined visual tools alongside standard web operations. The temporal tool selects an index $t$t to isolate an informative frame $v_t$vt​, and the spatial tool takes $v_t$vt​ and a bounding box $B \in \mathbb{R}^4$B∈R4 to crop a target entity, yielding a localized visual context $c_{vis} = \mathbf{Crop}(v_t, B)$cvis​=Crop(vt​,B) for downstream search.

To address the severe modality bias and parametric knowledge leakage observed in naive formulations, the authors introduce a comprehensive pipeline for data synthesis and agent training.

VQA Generation and Trajectory Construction The pipeline begins by synthesizing foundational Video QA pairs from a diverse collection of raw videos. After rule-based and agentic filtering, an agent-driven metadata curation process selects keyframes and localizes distinct entities. These entities are cropped and used to execute visual search queries, with a secondary model verifying semantic alignment. QA pairs are then synthesized using single-entity and multi-entity patterns. To guarantee that tasks strictly require external tool utilization, instances are rigorously filtered by conducting tool-free rollouts; if the agent answers correctly without tools, the instance is discarded.

Given the synthesized VQA pairs, the authors construct execution trajectories using a decoupled trajectory construction pipeline that explicitly separates visual perception from web exploration. To overcome modality bias, they employ a stage-wise tool unlocking strategy. In the initial phase, the agent is restricted to a vision-only action space comprising solely Select_Keyframe and Crop_Search, forcing extensive visual retrieval across multiple keyframes. Once the visual context is sufficient, the action space expands to include textual tools like Search and Visit to derive the final answer. This two-stage paradigm compels the model to conduct exhaustive cross-frame, cross-entity visual grounding prior to web exploration.

Multi-Modal Training Procedure The authors adopt a two-stage training paradigm to train the Video-DeepResearch agent. In the first stage, Supervised Fine-Tuning (SFT) is performed to establish a cold start and align the model with the decoupled perception-exploration workflow. Given a mixed dataset $\mathcal{D}$D where each instance consists of a context $x$x and target output sequence $y = \{y_1, \dots, y_N\}$y={y1​,…,yN​}, the SFT objective minimizes the standard autoregressive negative log-likelihood:

$$\mathcal{L}_{\mathrm{SFT}} = - \mathbb{E}_{(x, y) \sim \mathcal{D}} \left[ \sum_{i = 1}^{|y|} \log \pi_\theta(y_i \mid x, y_{<i}) \right]$$LSFT​=−E(x,y)∼D​​i=1∑∣y∣​logπθ​(yi​∣x,y<i​)​

To address the under-utilization of text tools, the training corpus is augmented with additional text-only QA instances.

In the second stage, Group Relative Policy Optimization (GRPO) is applied to refine the policy and incentivize endogenous exploration beyond static SFT imitation. GRPO computes advantages via intra-group relative rewards, eliminating the memory overhead of a separate value network. A sparse binary reward is applied, assigning $r = 1$r=1 for correct answers and $r = 0$r=0 otherwise. To prevent formatting violations or repetitive loops from dominating updates, the corresponding advantage is down-sampled with only a 20% probability. The GRPO objective is defined as:

$$\mathcal{L}_{\mathrm{GRPO}} = \frac{1}{G} \sum_{i=1}^{G} \left[ \min \left( \frac{\pi_\theta(o_i)}{\pi_{\mathrm{old}}(o_i)} \hat{A}_i, \operatorname{clip} \left( \frac{\pi_\theta(o_i)}{\pi_{\mathrm{old}}(o_i)}, 1 - \epsilon, 1 + \epsilon \right) \hat{A}_i \right) \right] - \beta \mathbb{D}_{\mathrm{KL}}$$LGRPO​=G1​i=1∑G​[min(πold​(oi​)πθ​(oi​)​A^i​,clip(πold​(oi​)πθ​(oi​)​,1−ϵ,1+ϵ)A^i​)]−βDKL​

Experiment

the paper evaluate vision-language models on the VIDEODR-BENCH and VideoDR tasks under an agentic setting with visual and text tools, using final-step predictions judged by a held-out model. The experiments demonstrate that a specialized training pipeline combining trajectory-synthesized visual grounding, text-augmented exploration, and reinforcement learning enables VIDEO-DEEPRESEARCH to substantially outperform proprietary models and corrects a pervasive modality bias toward parametric recall. Ablation studies confirm that each component is necessary for robust performance, and analysis shows that effective video deep research hinges on deliberate curriculum design rather than model scale alone.

In a naive setup, larger model scale does not guarantee better video deep research performance. GPT-5 matches the 397B-parameter Qwen model in accuracy while ignoring vision tools entirely, relying on parametric textual knowledge. A subsequent reinforcement learning phase pushes accuracy to 59.3%, demonstrating that training methodology outweighs raw scale. GPT-5 reaches 57% accuracy with zero vision tool invocations, indicating that benchmark performance can be achieved without genuine visual grounding. The 397B-parameter Qwen model attains only 58% accuracy and uses vision tools sparingly, showing that scale alone does not ensure robust video understanding. Reinforcement learning on a moderate-difficulty dataset improves accuracy to 59.3%, surpassing both the large Qwen and GPT-5 models.

VIDEODR-BENCH includes a balanced mix of short, medium, and long videos, with nearly half under two minutes. Reinforcement learning on a curated difficulty-focused dataset lifts benchmark accuracy by 2.0 percentage points to 56.5%, demonstrating that learned exploration strategies transfer across video lengths. The full training pipeline, combining visual grounding, textual deep research, and RL, reaches 59.3% overall, underscoring the necessity of autonomous exploration. Short videos (≤2 min) make up 46% of VIDEODR-BENCH, medium-length videos (2–10 min) 34%, and long videos (≥10 min) 20%, creating a temporally diverse evaluation. Applying GRPO reinforcement learning to a 2K moderate-difficulty dataset improves VIDEODR-BENCH performance by 2.0 percentage points, showing that the gained exploration behavior generalizes to benchmark-quality videos of varying durations. The final RL phase adds a further 2.5 percentage points over static imitation, pushing overall accuracy to 59.3% and confirming that autonomous exploration is critical for robust video deep research.

A specialized training pipeline that combines visual grounding, textual deep research skills, and reinforcement learning enables relatively compact models to set a new state of the art on video deep-research benchmarks. The 35B variant surpasses all proprietary models, while the 30B variant matches the top closed-source model and significantly outperforms a 397B open-source model, demonstrating that methodology decisively outweighs scale in this domain. The gains are especially pronounced in knowledge-intensive and entertainment categories, and the agent shifts from passive recall to active visual verification. The 35B model achieves the highest overall accuracy (64.0%) and leads on the VIDEODR-BENCH (65.4%), exceeding Claude-4.5-Sonnet by 5.0 percentage points. The 30B model attains 59.3% average accuracy, competitive with Claude-4.5-Sonnet and outperforming GPT-5 (52.5%) and Gemini 2.5 Pro (57.5%). Training methodology dominates model scale: a 30B model handily beats a 397B open-source model (52.8%) that lacks the specialized pipeline. Reinforcement learning and visual grounding yield large absolute gains of +20.3% in knowledge and +15.9% in entertainment categories over the foundation model. The agent's tool-use behavior transforms, with visual operations rising from 0.10 to 2.33 per task, signaling a shift from passive answering to active visual grounding.

Models on VideoDR often rely on parametric knowledge, with GPT-5 making zero visual tool calls and Qwen3.5-397B only 0.10. VideoDR-Bench enforces greater multimodal interaction, causing GPT-5's visual calls to increase to 0.31 and text calls to 1.43. The proposed VIDEO-DEEPRESEARCH-30B shifts to a balanced strategy, reaching 2.33 visual and 4.24 text calls on VideoDR and correcting the heavy text bias seen in baselines. On VideoDR, GPT-5 averages 0.00 visual tool calls while VideoDR-Bench raises this to 0.31, and text calls grow from 0.12 to 1.43 across the same datasets. Qwen3.5-397B exhibits a strong text bias with 0.10 visual and 1.27 text calls on VideoDR, whereas the proposed model attains 2.33 visual and 4.24 text calls, achieving a balanced multimodal search.

Starting from a baseline of 40.5% average accuracy, adding visual grounding trajectories (7K-SFT) lifts performance to 53.0%, with a larger absolute gain on VideoDR. Incorporating text-only deep research data further raises the average to 56.8%, and reinforcement learning with a moderate-difficulty set pushes it to 59.3%, revealing the complementary roles of perception, textual exploration, and self-driven training. Visual grounding SFT alone improves average accuracy by 12.5 points, with VideoDR jumping from 38.0 to 55.0 and VideoDR-Bench from 43.0 to 51.0. Adding 7K text-only QA instances yields a further 3.8-point average gain, directly mitigating the agent's tool-invocation bias. Applying GRPO on 2K moderate-difficulty samples provides an additional 2.5-point boost, especially benefiting VideoDR-Bench with a 2.0-point improvement.

Using VideoDR and VIDEODR-BENCH, which span diverse video lengths, the experiments demonstrate that larger model scale does not guarantee better video deep research performance; a carefully designed training pipeline that combines visual grounding, textual reasoning, and reinforcement learning allows compact models to surpass massive counterparts. The pipeline shifts agent behavior from passive reliance on parametric text knowledge toward active visual verification, with pronounced gains in categories requiring external knowledge and entertainment understanding. Overall, training methodology decisively outperforms raw scale, and autonomous visual exploration emerges as essential for robust video understanding.