One-sentence Summary

Nanjing University, Shanghai AI Laboratory, and colleagues present VideoChat3, a fully open video MLLM that pairs an Inflated 3D Vision Transformer (I3D-ViT) and Adaptive Frame for efficient spatiotemporal representation with a scalable data synthesis pipeline, achieving broad generalization that surpasses prior open-source models with only 4B parameters and enabling reproducible real-world video understanding.

Key Contributions

• VideoChat3 introduces an efficient architecture combining an Inflated 3D Vision Transformer (I3D-ViT) and adaptive frame resolution, which reduces the computational cost of spatiotemporal video processing during both training and inference.
• A scalable data synthesis pipeline constructs three diverse, high-quality datasets (VideoChat3-Academic2M, VideoChat3-LV116K, VideoChat3-OL617K) covering general, long-form, and streaming video scenarios, strengthening cross-domain generalization.
• The work releases complete training assets—model weights, code, strategy, and datasets—ensuring full reproducibility, and the 4B-parameter model surpasses prior open-source models of similar or larger sizes on general, long-form, and streaming benchmarks with higher efficiency.

Introduction

Video understanding is central to building general-purpose multimodal intelligence, as videos capture the continuous spatial and temporal dynamics that underpin applications from autonomous driving to real-time human-AI interaction. However, existing Video Multimodal Large Language Models (MLLMs) often specialize in narrow video settings, struggle with the computational cost of dense visual tokens in long or streaming videos, and suffer from a lack of open, reproducible training pipelines. The authors address these issues with VideoChat3, a systematic framework that pairs an efficient Inflated 3D Vision Transformer with adaptive frame resolution to drastically reduce token overhead, while a scalable data curation pipeline produces three high-quality instruction datasets for offline and streaming tasks. By fully open-sourcing the model weights, training code, data, and construction pipeline, VideoChat3 provides a strong, reproducible foundation for generalist video understanding that outperforms comparable open-source models.

Dataset

The dataset built for VideoChat3 combines three complementary subsets, each tailored to a specific supervision challenge: dense instruction following on short clips, reasoning over long videos, and proactive streaming response.

• VideoChat3-Academic2M (approx. 2 million samples)

• Sources: Aggregated from publicly available academic video datasets spanning captioning, question answering, and fine-grained motion perception. These datasets offer broad task coverage and human-verified labels.

• Processing: Original annotations are often short (multiple‑choice letters, noun phrases, short factual answers). The authors use Qwen3‑VL‑235B to rewrite them into evidence‑grounded, comprehensive responses while preserving the original answer semantics. A consistency verification stage then compares each rewritten response with the original label; only samples that remain semantically consistent are kept. This yields dense targets that explicitly describe visual evidence and reasoning logic.

• Output: Enhanced instruction‑response pairs with richer video‑specific information, still bounded by the reliability of the original academic labels.

• VideoChat3-LV116K (116.2 K rows)

• Sources: Newly collected long videos from diverse domains (entertainment, education, sports, news, science, gaming). Videos with severe corruption, low visual quality, excessive duplication, or limited semantic content are discarded.

• Processing:

• Segmentation: Each video is split into coherent temporal units with PySceneDetect, combining shot detection and duration constraints to avoid overly short or long segments.

• Segment annotation: Qwen3‑VL‑235B generates structured descriptions (entities, actions, scenes, event transitions, OCR cues) for every segment. A quality filter removes repetitive, generic, or inconsistent descriptions.

• Evidence ledger: The validated, timestamped segment descriptions form a textual evidence ledger for the whole video.

• Training synthesis: A frontier LLM consumes the interleaved timestamps and segment descriptions to produce supervision for three task families: temporal grounding (single‑ and multi‑interval), video timelines, and long‑video QA. QA samples are filtered to exclude questions answerable from a single frame, a single segment, or language priors alone.

• Result: 116.2 K JSONL rows with mean durations from 156 s to about 1.3 K seconds per shard, providing long‑range supervision that requires cross‑segment aggregation and event‑level reasoning.

• VideoChat3-OL617K (617 K samples)

• Sources: High‑quality video‑question‑answer triples.

• Processing:

• Visual clue localization: A vision‑language model identifies temporal intervals that contain the key evidence needed to answer the query.

• Clue verification: The candidate clue interval is cropped from the original video. The model re‑evaluates the cropped segment; only samples where the answer can be correctly inferred from the cropped clip alone are retained.

• Streaming construction: Verified clue intervals are treated as moments to collect evidence. The corresponding video windows are labeled with a </Standby> token. After the clue interval ends, the model must emit </Response> followed by the answer. All other windows receive a </Silence> token, teaching the model to suppress premature responses.

• Output: Interleaved streaming sequences of video windows and response‑state tokens, turning offline QA supervision into proactive online QA data with explicit evidence‑acquisition and response‑timing signals.

How the data is used All three subsets are combined during training to teach VideoChat3 multiple capabilities. VideoChat3-Academic2M supplies dense, evidence‑aware instruction‑following examples on short clips and diverse tasks. VideoChat3-LV116K extends the temporal context to minutes or hours, forcing the model to localize events, aggregate information across segments, and reason over sparse evidence. VideoChat3-OL617K trains the model to act as a streaming assistant, monitoring a video stream and responding at the precise moment when sufficient visual evidence has accumulated. The datasets are used jointly; no explicit mixture ratios are reported.

Method

The authors address the fundamental tension between visual perception quality and computational efficiency in Video MLLMs by introducing VideoChat3, a unified video-oriented perception architecture. The framework is built on the principle that the spatiotemporal structure of videos should be modeled and redundant information reduced as early as possible before visual tokens are passed to the language model.

To achieve efficient video tokenization with spatiotemporal modeling, the authors introduce the Inflated 3D Vision Transformer (I3D-ViT). Instead of treating frames as isolated images, I3D-ViT inflates a pretrained image tokenizer by extending its 2D spatial self-attention into 3D spatiotemporal self-attention.

As shown in the figure above, the visual tokenizer partitions each video into contiguous frame chunks of up to $T$T frames. Within each chunk, the model preserves pretrained spatial positional embeddings and introduces learned absolute temporal embeddings for frame indices. Tokens from all frames within a chunk are flattened into a single sequence, enabling the I3D-ViT blocks to perform joint attention over space and time at the native input resolution. After spatiotemporal contextualization, chunk-wise temporal pooling aggregates features along the temporal dimension, reducing the token count by a factor of $T$T. Combined with spatial downsampling, this yields a high spatiotemporal compression ratio while preserving local temporal dynamics.

For streaming video perception, the authors design an Adaptive Frame Resolution mechanism inspired by human visual attention. The model formulates streaming inference as a state-conditioned closed loop between the language model and the visual tokenizer.

As illustrated in the diagram, at each streaming step, the model emits a response-state token belonging to one of three states: Silence, Standby, or Response. This state token acts as a control signal for the next visual input. A deterministic controller adjusts the per-frame pixel quota $b_{t+1}$bt+1​ based on the predicted state $s_t$st​:

$$b_{t+1} = \begin{cases} B_{\text{low}} & \text{if } s_t = \text{Silence}, \\ B_{\text{high}} & \text{if } s_t = \text{Standby}, \\ B_{\text{low}} & \text{if } s_t = \text{Response}, \end{cases}$$bt+1​=⎩⎨⎧​Blow​Bhigh​Blow​​if st​=Silence,if st​=Standby,if st​=Response,​

where $B_{\text{low}} = 224^2$Blow​=2242 pixels and $B_{\text{high}} = 448^2$Bhigh​=4482 pixels. This allows the model to maintain low-cost monitoring during routine intervals and allocate higher visual fidelity when potential evidence appears.

To construct high-quality training data, the authors treat academic datasets as reliable semantic anchors and enhance them into richer instruction-following responses.

As demonstrated in the example, short or option-only answers are rewritten into comprehensive responses with explicit visual evidence and reasoning using a large vision-language model. A consistency verification stage then filters out rewritten responses that contradict or over-specify the original labels, ensuring the supervision remains faithful to the ground truth while providing denser training signals.

For long-video understanding, the authors build a dedicated synthesis pipeline to construct structured annotations.

The pipeline first filters candidate long videos and decomposes them into manageable temporal segments using visual scene detection. Each segment is independently annotated to capture visible entities, actions, and event transitions. An auxiliary model evaluates these segment descriptions to discard repetitive or unsupported annotations. Finally, the validated segment descriptions are aggregated into dense, timestamped captions for the full video, which are then used to synthesize training labels for temporal grounding, timelines, and long-video QA tasks.

The overall training recipe proceeds through four stages. In Stage-0, the authors pre-train the visual tokenizer in a language-grounded manner. They initialize I3D-ViT from an image-pretrained model, attach a lightweight projector, and use a temporary text decoder. This stage consists of a projector warm-up followed by full-parameter fine-tuning on a large mixture of image-text and video-text pairs. In Stage-1, the temporary decoder is replaced with the target language model for video-language alignment, again following a lightweight-to-joint schedule. Stage-2 converts the aligned model into a general-purpose video instruction-following model by training on a balanced mixture of image and video QA and caption samples. Finally, Stage-3 adapts the model to long-form and streaming video understanding. For streaming behavior, the authors introduce a state transition masking strategy to balance the loss over state tokens. Let $s_t$st​ be the target state at streaming step $t$t. The loss is computed over transition positions $\mathcal{T} = \{t \mid s_t \neq s_{t-1}\}$T={t∣st​=st−1​} and a uniformly sampled subset $\widetilde{\mathcal{C}}$C of continuation positions $\mathcal{C} = \{t \mid s_t = s_{t-1}\}$C={t∣st​=st−1​} such that $|\widetilde{\mathcal{C}}| = |\mathcal{T}|$∣C∣=∣T∣. The masked state loss is:

$$\mathcal{L}_{\text{state}} = - \frac{1}{\sum_t m_t} \sum_t m_t \log p_\theta(s_t \mid V_{\leq t}, y_{<t}), \quad m_t = \mathbb{1}[t \in \mathcal{T} \cup \widetilde{\mathcal{C}}].$$Lstate​=−∑t​mt​1​t∑​mt​logpθ​(st​∣V≤t​,y<t​),mt​=1[t∈T∪C].

This objective forces the model to compare the previous state with current visual evidence before deciding whether to maintain or change its behavior, effectively learning both the temporal response policy and the active visual-budget policy.

Experiment

The evaluation spans general video understanding, streaming perception, efficiency, and ablation studies. VideoChat3-4B achieves strong temporal perception and grounding, matching or surpassing similarly sized open models and competing with large proprietary systems, while its online variant extends these capabilities to proactive streaming, accurately deciding when to respond. The architecture's early spatio-temporal compression reduces visual tokens, shifting computation from the quadratic LLM stage to the linear encoder and yielding substantial latency and memory savings for long videos. Ablation studies further confirm that the I3D-ViT encoder, dynamic streaming components, and enhanced training data each contribute meaningfully to the model's offline and online performance.

VideoChat3 training proceeds through four stages that progressively extend packed sequence length from 8K to 98K tokens while decreasing learning rates. The visual tokenizer is pre-trained first with a temporary language model that is later discarded, and the largest dataset (10.33M samples) is used for video instruction tuning. Trainable modules shift from projector-only warm-up to full model, and finally to projector and LLM for streaming adaptation. Stage 0 pre-trains the visual tokenizer with a temporary LLM that is discarded afterward, using the highest learning rate (1e-3) and batch size (512). Packed sequence length increases across stages from 8,192 tokens in Stage 0 to 98,304 tokens in Stage 3, supporting long-form and streaming video understanding.

VideoChat3-4B, a fully open model, achieves strong and balanced performance across temporal perception, long-video understanding, reasoning, and temporal grounding. It outperforms or matches similarly sized fully open models and surpasses several proprietary models on temporal grounding, while improving on 18 of 19 metrics over Qwen3-VL-4B. The most notable gains appear in temporal grounding tasks, where it significantly exceeds VideoChat-Flash-7B and proprietary counterparts like GPT-5 and Gemini 2.5 Flash. VideoChat3-4B obtains the best fully open results on MotionBench (61.7) and TempCompass (75.6), demonstrating strong motion and temporal dynamics understanding. Compared to Qwen3-VL-4B, VideoChat3 improves on 18 of 19 directly comparable metrics, with the only decrease on the open-ended split of VideoEval-Pro. Temporal grounding gains over VideoChat-Flash-7B are substantial, reaching +16.4/+29.8/+34.3 on TimeLens splits, +30.6/+30.1 on VUE-TR v1/v2, and +18.7 on MomentSeeker. VideoChat3 surpasses GPT-5 and Gemini 2.5 Flash on all three TimeLens splits and remains competitive with Gemini 2.5 Pro.

VideoChat3-4B effectively extends its offline video understanding to online streaming, achieving top perception‑and‑memory results on four of six benchmarks and strong proactive response timing. It outperforms specialized models without auxiliary modules and improves over a comparable baseline on nearly all metrics, demonstrating continuous perception, context retention, and timely response as videos unfold. It leads on four of six perception‑and‑memory benchmarks, with a 12.4‑point gain on ODVBench over StreamForest and consistent margins of 0.9–2.9 points on OVOBench, StreamingBench, and River. On proactive response timing, it achieves an average F1 of 35.5 on OVO‑Timing, surpassing the specialized Em‑Garde model by 4.5 points without using an auxiliary 2B module. Compared to Qwen3‑VL‑4B under the same evaluation settings, VideoChat3 improves on 10 of 11 directly comparable metrics, with especially large leaps in online timing and the TV subset of ProactiveVQA. It beats Qwen3‑VL‑4B on three of four ProactiveVQA subsets, though it remains behind the specialized MMDuet‑2 model.

VideoChat3 introduces spatio-temporal modeling in the vision encoder, which increases encoder latency but halves the number of visual tokens passed to the LLM. This trade-off shifts computation from the quadratically scaling LLM to the linearly scaling encoder, reducing total latency, FLOPs, and GPU memory for long videos. The benefit grows with video length, making VideoChat3 substantially more efficient for long-video understanding. VideoChat3 consistently produces only half as many visual tokens as Qwen3-VL for the same number of input frames, from 256 to 1024 frames. At 2048 frames, VideoChat3 reduces total latency by more than half (from 44.4s to 20.4s) and cuts FLOPs by over 60%, while also lowering GPU memory by 26.14 GB.

I3D-ViT, initialized from MoonViT weights, delivers comparable image understanding performance with a marginal average improvement. On video tasks, it consistently and significantly outperforms MoonViT, a gain attributed to its 4× temporal compression that allows nearly four times more sampled frames during training and inference. I3D-ViT matches or slightly exceeds MoonViT on image benchmarks, with an average score of 66.10 versus 65.30. I3D-ViT substantially surpasses MoonViT across all video task categories, leveraging its higher frame sampling capacity.

VideoChat3 is trained in four stages with progressively longer packed sequences and evaluated on offline video understanding, online streaming, and efficiency benchmarks. It achieves strong balanced performance, surpassing similarly sized open models and proprietary systems on temporal grounding, while extending effectively to online streaming with top perception and memory results. The model's spatio-temporal encoder halves visual tokens, significantly reducing latency and memory for long videos, and its I3D-ViT initialization yields substantial video gains through higher frame sampling.