One-sentence Summary

Researchers from Nanjing University, Shanghai AI Laboratory, and colleagues propose TimeLens2, a generalist video temporal grounding model that treats evidence as interval sets and uses a multi-span annotation pipeline with a temporal Wasserstein reward—exact $1D$1D $W_1$W1​ distance between uniform distributions over merged intervals—to achieve state-of-the-art performance across seven benchmarks, improving Qwen3-VL backbones by up to 18.1 mIoU.

Key Contributions

• TimeLens2-93K provides reliable single- and multi-span supervision for long, diverse videos by combining caption-derived proposals, independent localization, cross-agent consensus, semantic verification, and boundary refinement.
• A temporal Wasserstein reward computes the exact one-dimensional W₁ distance between uniform distributions over merged interval supports, delivering dense, matching-free feedback for set-valued predictions, and is complemented by temporal IoU for precise overlap.
• Across seven benchmarks, TimeLens2 models (2B, 4B, 8B) outperform all size-matched baselines; the 4B and 8B variants achieve state-of-the-art performance, surpassing open-source models up to 397B parameters, and improve over their Qwen3-VL backbones by 13.0–18.1 mIoU points.

Introduction

Video multimodal large language models (MLLMs) can describe what happens in a video, but users also need to know when the supporting evidence appears; without precise temporal intervals, outputs remain unverifiable. Temporal grounding, the task of localizing single or multiple evidence intervals across diverse video lengths, query forms, and viewpoints, is therefore essential for making MLLM outputs traceable. Prior work faces two structural mismatches: long-video supervision often relies on a single global annotation that misses repeated evidence and produces imprecise boundaries, while optimization with temporal IoU rewards assigns zero to all disjoint predictions and cannot distinguish near misses from distant errors, especially for multi-span targets. The authors introduce TimeLens2, a generalist temporal-grounding MLLM that treats evidence as a first-class interval set throughout training. They first build TimeLens2-93K, a staged evidence-verification pipeline that replaces brittle one-pass annotation with hierarchical captioning, independent relocalization, consensus, and boundary refinement to yield reliable multi-span supervision. They then propose a temporal Wasserstein reward that computes the exact one-dimensional Wasserstein distance between merged predicted and target interval supports, providing dense, matching-free geometry feedback for disjoint and unequal-cardinality predictions. Compact TimeLens2 models trained with this approach outperform much larger open-source models across seven benchmarks spanning long-video, multi-span, question-form, and egocentric grounding.

Dataset

The authors construct TimeLens2-93K, a large-scale temporal grounding dataset designed to teach models to search for supporting evidence in long, diverse videos. The dataset consists of 23,793 videos and 93,232 grounding instances, each represented as a video, a declarative query, and a variable-cardinality set of supporting time intervals. Among these, 12,091 instances contain multiple supporting spans, unifying single- and multi-span grounding in a full-video context.

Data sources and stratification

• The video pool is drawn from 34,867 YouTube videos, deliberately stratified across five duration ranges (under one minute to over one hour) and diverse visual domains.
• Duration controls the search horizon and distractor load, while domain breadth ensures varied evidence types.

Construction pipeline The pipeline separates candidate proposal from label determination through a staged factorization:

• Hierarchical temporally grounded captions: Each video is partitioned into semantically coherent clips of 20–60 seconds using PySceneDetect. A large vision-language model (Qwen3-VL-235B-A22B) then generates a global caption and per-clip segment captions that emphasize observable events and temporal order. This hierarchical description serves as a semantic index.
• Segment-aware query synthesis: Another model (Kimi-K2.5) jointly produces a non-redundant declarative query and selects all supporting segments from the caption hierarchy, forming a coarse multi-span proposal.
• Dual-agent local grounding: Two independent models (Qwen3-VL-30B-A3B and TimeLens-8B) relocalize each query-proposal pair directly from the video clip at one-second resolution, returning one or more intervals or an empty set. Empty outputs reject unsupported proposals; multi-interval outputs preserve repeated evidence. Clip-relative timestamps are mapped to the original video timeline.
• Cross-agent consensus and semantic verification: Instances are retained only if the two agents’ merged interval sets achieve a set-level IoU > 0.9, and if the normalized text–video cosine similarity (via Qwen3-VL-Embedding) between the query and the retained clip is at least 0.5. This ensures temporal reproducibility and query–evidence alignment.
• Boundary-focused local refinement: For each surviving boundary, a strong model observes a ±3-second neighborhood and predicts a refined transition point. Adjacent spans separated by at most one second are merged, treating such gaps as boundary jitter rather than semantic breaks.

Dataset usage TimeLens2-93K is used to train temporal grounding models that must locate evidence within full-length videos. The variable-cardinality interval formulation allows models to learn both single- and multi-span retrieval from long, distraction-rich contexts. The construction process provides high-confidence interval-set labels through a cascade of global context proposals and concentrated local verification.

Method

The authors address the fundamental supervision challenge in long videos by constructing the TimeLens2-93K dataset and developing a corresponding training framework for generalist video temporal grounding. The data construction pipeline separates candidate construction from label determination to ensure reliable annotations.

As shown in the figure below:

The pipeline begins with a long and diverse video source pool, stratifying videos across five duration ranges to control search difficulty and distractor load. To avoid conflating video understanding, query synthesis, and localization, the authors convert each video into a hierarchical, time-stamped description. They partition videos into semantically coherent clips using content-based boundaries and generate global and segment captions using Qwen3-VL-235B-A22B. These captions serve as a semantic index. For segment-aware query synthesis, Kimi-K2.5 conditions on the captions to jointly generate declarative queries and select supporting segments, forming coarse proposals.

To convert these coarse text matches into verifiable visual decisions, the authors employ dual-agent local grounding. Qwen3-VL-30B-A3B and TimeLens-8B independently relocalize each proposal from short video clips. Cross-agent consensus and semantic verification then filter the results. Temporal reproducibility is checked using a set-level IoU metric:

Instances with $\mathrm{IoU}_{\text{set}} > 0.9$IoUset​>0.9 are retained, and semantic validity is verified by requiring a normalized text-video cosine similarity of at least 0.5. Finally, boundary-focused local refinement sharpens the surviving boundaries. Qwen3-VL-235B-A22B observes a local neighborhood around each boundary to predict refined transition points, and adjacent spans separated by at most one second are merged.

For the model training process, the authors separate capability acquisition from geometric calibration. Long-context supervised grounding fine-tunes Qwen3-VL on examples from TimeLens2-93K, TimeLens-100K, and Ego4D-NLQ. Retaining full-video context forces the model to isolate sparse evidence among competing events. To ensure the model learns protocol-invariant interval semantics, the authors introduce instruction and response-format diversity, rendering the same ground truth in diverse single- or multi-span formats.

Reinforcement learning is then used to directly optimize decoded interval sets. The authors implement rollout-guided hard-sample mining, where examples with lower mean tIoU scores from the SFT checkpoint receive higher sampling weights during GRPO, creating an adaptive curriculum.

To address the limitations of standard tIoU rewards, which suffer from overlap plateaus and matching artifacts, the authors introduce a temporal Wasserstein reward.

Refer to the framework diagram:

The total reward combines set-level tIoU, the temporal Wasserstein reward $R_{\mathrm{TW}}$RTW​, and a penalty for invalid outputs:

While tIoU measures recovered support, it lacks geometric guidance. The authors lift each interval set to a uniform temporal distribution over its merged support:

The 1-Wasserstein distance measures the transport needed to align the predicted and target mass:

This distance is converted into a scale-normalized similarity:

This formulation ensures the reward depends on where evidence lies rather than how the support is partitioned, providing dense, partition-invariant credit for multi-span grounding.

Experiment

TimeLens2 is evaluated on seven temporal grounding benchmarks using mIoU and Recall@1 at tIoU 0.5, with models trained via supervised fine-tuning on a curated dataset and reinforcement learning with a temporal Wasserstein reward. The model consistently outperforms prior open-source and proprietary systems, especially on long videos and diverse query forms, demonstrating evidence-directed temporal search rather than superficial boundary fitting. Ablations show that the TimeLens2-93K dataset teaches a transferable search capability, progressive label curation removes annotation errors, and the temporal Wasserstein reward provides dense distance-aware feedback that improves recovery from near misses and breaks ties in group-relative optimization.

TimeLens2 achieves consistent gains across model scales and temporal grounding benchmarks, outperforming size-matched open-source baselines and often surpassing larger proprietary systems. The smallest variant, TimeLens2-2B, already exceeds all prior 2B open-source models and the previous 8B model, while the 4B version beats every prior model on six of seven benchmarks. Improvements are largest on challenging multi-interval and question-form tasks, where the method adds up to 27.8 points in mIoU over its backbone. TimeLens2-2B outperforms all three size-matched open-source baselines on all seven benchmarks, reaching 44.5 average mIoU — 14.2 points above Qwen3-VL-2B and 2.4 points above the prior 8B model. TimeLens2-4B surpasses every prior model, including proprietary systems, on six of seven benchmarks, with mIoU gains of 19.6 points on VUE-TR and 27.8 on VUE-TR-V2 over the Qwen3-VL-4B backbone.

Scaling TimeLens2-93K exhibits a two-phase pattern: the first 5% of data provides the largest average mIoU jump from 34.7 to 42.8, while further increases up to 50% add only modest gains, plateauing around 45.3. The full corpus (45.8) mainly improves harder settings like VUE-TR-V2 and Ego4D-NLQ, and the learned search transfers across query forms, boosting question-style MomentSeeker from 15.3 to 25.8 despite training on declarative queries. The initial 5% of TimeLens2-93K raises average mIoU by 8.1 points, accounting for most of the total improvement. After 20% of the data, average mIoU saturates at 45.3, with subsequent scaling only benefiting long-context and ambiguous-boundary benchmarks.

Progressive label curation demonstrates a quality-over-quantity pattern: filtering by temporal consensus and semantic verification shrinks the training set to about one-eighth of its original size while raising average mIoU from 42.0 to 44.1, and subsequent boundary refinement on the same reduced set adds the largest stage gain of 1.7 points, reaching 45.8. The full cascade improves over raw Qwen labels by 3.8 points overall, with particularly large boosts on ActivityNet and QVHighlights, though minor regressions occur on VUE-TR and MomentSeeker. Temporal consensus and semantic verification cut the label pool from over 735K to 93.2K QA turns while lifting average mIoU from 42.0 to 44.1, showing that removing unstable or irrelevant labels improves performance even with far fewer examples. Boundary refinement on the already-filtered labels yields the largest single-stage gain (+1.7 mIoU), indicating that boundary noise becomes the dominant bottleneck once temporal reproducibility and semantic relevance are ensured.

Increasing the maximum packed length from 16K to 100K tokens raises the average moment retrieval mIoU from 44.4 to 46.4, with the largest gain coming at 32K. The improvement is concentrated on long-video and egocentric benchmarks like VUE-TR-V2 and Ego4D-NLQ, while shorter-context tasks show only marginal changes. Expanding the packed length from 16K to 32K lifts the average mIoU by 1.0 point, primarily from VUE-TR-V2 (+2.6) and Ego4D-NLQ (+2.1). Beyond 32K, gains diminish: 64K adds 0.5 points and 100K adds another 0.5 points, with most datasets reaching near-plateau performance.

Diversifying both the instruction phrasing and the response format for temporal grounding improves performance over a fixed-template baseline. Using diverse response formats alone provides a clear gain, while diversifying instructions alone offers a marginal boost. The fully diverse setting achieves the highest average mIoU, outperforming the baseline across all datasets. Diversifying the response format alone yields a noticeable improvement over the fixed baseline, whereas diversifying the instruction phrasing alone gives almost no gain. Pairing diverse instructions with diverse response formats leads to the best overall performance, with consistent gains across all evaluated datasets.

TimeLens2 demonstrates consistent improvements across model scales and temporal grounding benchmarks, with even compact variants outperforming larger models on challenging multi-interval and question-based tasks. Data scaling follows a saturating pattern where the initial 5% of training data provides the majority of gains, while progressive label curation reveals that quality filtering through temporal consensus and semantic verification yields better performance than raw data quantity. Longer packed sequence lengths primarily benefit long-video and egocentric benchmarks, and diversifying both instruction phrasing and response formats during training leads to the most robust overall performance.