One-sentence Summary

NJU et al. propose VideoJev, a System One model family that replaces autoregressive decoding with a shared head scoring temporal intervals and textual options to unify temporal grounding and multiple-choice video QA, employs Cached and Joint inference for efficient visual reuse, and achieves 6.7 to 10.8 point mIoU gains over backbones with measured speedups of 2.8 to 3.1× for temporal grounding and over 100× for video QA.

Key Contributions

• VideoJev introduces a family of video System One models that replace autoregressive answer decoding in pretrained multimodal large language models with a shared scoring head for temporal intervals and textual options, unifying temporal grounding and multiple-choice video QA as direct candidate scoring.
• The models reuse visual computation across repeated queries through Cached inference for arriving queries and Joint inference for multiple queries processed together, supporting efficient structured judgments over the same visual history.
• Across three temporal grounding benchmarks and four video QA benchmarks, VideoJev-4B and VideoJev-35B-A3B improve mean mIoU over their backbones by 6.7 and 10.8 points with 2.8 to 3.1 times inference speedups on temporal grounding, while achieving QA performance comparable to their backbones in Thinking mode with over 100 times speedups.

Introduction

Video understanding for agents requires timely, structured decisions over shared visual history, such as verifying whether an ingredient was added before choosing the next cooking step. Prior work offers complementary but incomplete strengths: discriminative vision models make efficient predictions yet remain task-specific or tied to fixed label spaces, while video multimodal large language models provide flexible semantic understanding but rely on autoregressive generation even for compact outputs like temporal boundaries or multiple-choice answers. The authors introduce VideoJev, a family of video System One models that unifies temporal grounding and multiple-choice video question answering through direct candidate scoring. It reuses visual computation with cached inference for arriving queries and joint inference for batched queries, and the authors evaluate dense 4B and mixture-of-experts 35B-A3B models across seven benchmarks, reporting improved grounding and QA performance with substantial inference speedups.

Method

The authors propose VideoJev, a unified video decision framework that scores temporal intervals and textual options using a shared head over pretrained multimodal representations. As shown in the figure below, the architecture adapts a pretrained multimodal large language model with a causal language backbone and native video inputs.

To leverage the semantic similarity supported by pre-MLP features, the authors remove the final MLP and replace the language-model output head with a candidate scorer. They introduce two new tokens, <|option_end|> and <|decide|>, which provide candidate readouts and a query-level decision readout, respectively. Both tokens are initialized from the pretrained <|im_end|> embedding. Separate linear projections map their hidden states to 512 dimensions and normalize them to unit length:

$$u = \frac{W_q h_d + b_q}{\|W_q h_d + b_q\|_2}, \quad v_i = \frac{W_k h_i + b_k}{\|W_k h_i + b_k\|_2}, \quad s_i = 10 u^\top v_i + b$$u=∥Wq​hd​+bq​∥2​Wq​hd​+bq​​,vi​=∥Wk​hi​+bk​∥2​Wk​hi​+bk​​,si​=10u⊤vi​+b

where $h_d$hd​ and $h_i$hi​ are the decision and candidate states, $W_q, W_k, b_q, b_k$Wq​,Wk​,bq​,bk​ are learned projection parameters, and $\|\cdot\|_2$∥⋅∥2​ is the Euclidean norm. The normalized similarity yields the logit $s_i$si​, scaled by a fixed factor of 10 and a learned scalar bias $b$b.

Given a video $V$V, a language query $q$q, and $K$K candidates $C = \{c_1, \ldots, c_K\}$C={c1​,…,cK​}, the model places video tokens first, followed by the task instruction, query, and candidate list. Each candidate ends with <|option_end|>, and the list ends with <|decide|>. Causal modeling allows candidate readouts to utilize the video, query, and preceding candidates, while the decision readout processes the full list. For temporal grounding, candidates are timestamp texts identifying frame-pair intervals, and their sigmoid scores indicate event membership to decode a contiguous span. For multiple-choice video QA, candidates are answer texts, and softmax normalizes their logits into probabilities to select the highest-scoring option.

To optimize computational efficiency, the authors design reusable video computation. Since causal modeling makes the video prefix independent of subsequent queries, its computation can be shared. In cached inference, the video is encoded and prefilled once, and each arriving query branches from the cached prefix. In joint inference, multiple queries and their candidate lists follow a single video prefix, with a separate <|decide|> readout for each query, allowing one causal language-model forward pass to yield all query scores.

For temporal grounding, the timeline is partitioned into $K$K candidate intervals. The authors train local membership and coverage supervision to identify relevant intervals and cover the event as a whole. They employ multi-label binary cross-entropy to supervise each candidate independently:

$$\mathcal{L}_{\mathrm{BCE}} = - \frac{1}{K} \sum_{i=1}^{K} \left[ y_i \log p_i + (1 - y_i) \log (1 - p_i) \right]$$LBCE​=−K1​i=1∑K​[yi​logpi​+(1−yi​)log(1−pi​)]

where $y_i$yi​ is the ground truth label and $p_i = \sigma(s_i)$pi​=σ(si​) is the membership probability. To directly reward event coverage, they add a soft intersection-over-union loss over the positive intervals:

$$\mathcal{L}_{\mathrm{IoU}} = 1 - \frac{\sum_i d_i p_i y_i}{\sum_i d_i [y_i + p_i (1 - y_i)]}$$LIoU​=1−∑i​di​[yi​+pi​(1−yi​)]∑i​di​pi​yi​​

where $d_i$di​ is the interval duration. To decode a continuous event span from interval scores, the authors adjust logits by a probability threshold $\tau$τ and select the nonempty contiguous range with the largest sum using Kadane's algorithm in $O(K)$O(K) time:

$$(a^\star, b^\star) = \arg\max_{1 \leq a \leq b \leq K} \sum_{i=a}^{b} \left[ s_i - \log \frac{\tau}{1 - \tau} \right]$$(a⋆,b⋆)=arg1≤a≤b≤Kmax​i=a∑b​[si​−log1−ττ​]

For multiple-choice video QA, the shared head assigns a logit $s_i$si​ to each option. Since exactly one answer is correct, the authors normalize the logits with softmax and train with cross-entropy:

$$p_i = \frac{\exp(s_i)}{\sum_{j=1}^{K} \exp(s_j)}, \qquad \mathcal{L}_{\mathrm{CE}} = - \log p_y$$pi​=∑j=1K​exp(sj​)exp(si​)​,LCE​=−logpy​

where $y$y is the index of the correct option. To reduce position bias, a reproducible, answer-independent permutation is applied to each eligible training question.

The training recipe follows a two-stage process. In the first stage, the authors train only the new scorer against fixed pretrained features on 30 percent of the videos, keeping the backbone and new marker embeddings frozen. The grounding and QA objectives are simply the binary cross-entropy and cross-entropy losses. In the second stage, they train the language backbone, marker embeddings, and aligned head on the remaining 70 percent of the videos. The grounding objective incorporates the soft intersection-over-union loss for event-level coverage supervision, while the QA objective retains cross-entropy. Task losses are averaged equally over questions, with the visual encoder and merger remaining frozen throughout.

Experiment

The experiments evaluate VideoJev on video temporal grounding and multiple-choice video QA, covering benchmarks such as Charades-STA, ActivityNet Captions, QVHighlights, LVBench, Video-MME, LongVideoBench, and MLVU. VideoJev improves grounding accuracy and localization precision while reducing inference latency compared with generative baselines, and it also raises QA scores over its instruction-tuned backbones while remaining competitive with reasoning models at much higher speed. Ablations confirm that the gains come from the architectural readout rather than data alone, that joint inference and joint grounding plus QA training are beneficial, and that auxiliary Soft IoU supervision helps grounding and QA. The analysis further shows that scoring many queries jointly scales efficiently and that model confidence tracks accuracy but tends to be overconfident.

VideoJev models report consistent temporal grounding gains over generative and TimeLens-style baselines, with improvements concentrated at stricter overlap thresholds. The scoring architecture also yields the lowest measured latency among open-source grounding systems and several-fold speedups over generative backbones. Scaling from the 4B model to the 35B-A3B model adds only a small quality gain and a modest latency increase. VideoJev improves mean mIoU and R1 at stricter overlap thresholds relative to its generative backbones and TimeLens-style baselines. The larger VideoJev model adds a small mean mIoU gain while increasing measured latency only modestly. VideoJev models are several times faster than their generative backbones and have the lowest latency among open-source grounding models. Among listed open-source dense baselines, Qwen3-VL-4B outperforms Molmo2-4B across all three grounding datasets.

VideoJev models consistently outperform their instruct backbones on multiple-choice video QA, with gains on every evaluated benchmark and particularly strong LVBench results. Direct scoring is competitive with explicit reasoning models, surpassing them on LVBench and LongVideoBench while trailing on Video-MME and MLVU, and provides large latency reductions. Candidate confidence tracks accuracy, suggesting selective trust or escalation is feasible. VideoJev-4B and VideoJev-35B-A3B improve four-benchmark mean scores over their instruct backbones, with improvements on all QA benchmarks. Both VideoJev variants lead their open-source groups on LVBench, and the MoE model achieves the highest group mean. Direct scoring matches or exceeds reasoning counterparts on LVBench and LongVideoBench, while reasoning retains an edge on Video-MME and MLVU. Measured speedups over thinking baselines are roughly two orders of magnitude. Higher option probability is associated with higher accuracy, enabling more reliable high-confidence predictions.

When fine-tuned on identical post-training data, VideoJev-4B attains higher average temporal grounding and video QA scores than an autoregressive Qwen3.5-4B control. The control only leads on Charades temporal grounding, while VideoJev-4B shows the clearest advantages on QVHighlights, LongVideoBench, and MLVU. Matched data and schedules support an architectural benefit from directly supervising structured decisions. VideoJev-4B improves mean grounding and QA performance over the autoregressive control under the same training data and schedule. The autoregressive control trails most on QVHighlights, LongVideoBench, and MLVU, with its only temporal grounding lead on Charades.

Joint inference generally outperforms cached inference on temporal grounding and video QA across both VideoJev model sizes, improving 13 of 14 model-benchmark comparisons. The only exception is a small cached-inference edge on QVHighlights for the larger model. Efficiency measurements indicate these quality gains come with only a small marginal cost as the number of queries grows. Joint inference improves mean grounding and QA scores over cached inference for both the 4B and 35B-A3B models. The benefit is broad, with 13 of 14 model-benchmark comparisons favoring joint inference and only one slight cached-inference advantage on QVHighlights for the larger model.

Joint training of VideoJev-4B on temporal grounding and video question answering improves both task families relative to training on either task alone. Temporal grounding performance rises across all three grounding benchmarks, and QA performance rises across all four QA benchmarks. The results indicate that the two tasks reinforce rather than interfere with each other. Joint training outperforms single-task training on every grounding and QA benchmark. Video QA benefits more than temporal grounding, with the largest QA improvement on LongVideoBench.

VideoJev is evaluated on temporal grounding and multiple-choice video question answering against generative, TimeLens-style, and autoregressive baselines. Direct structured scoring yields consistent grounding gains at stricter overlaps with lower latency, improves video QA over instruction backbones with confidence tracking accuracy, and shows architectural benefits over an autoregressive control trained on matched data. Joint training and joint inference further improve both task families, and scaling from the 4B to the 35B-A3B model adds only modest gains.