One-sentence Summary

Researchers from Sejong University and NAVER Cloud introduce Video-Oasis, a sustainable diagnostic suite that systematically audits video understanding benchmarks, revealing that $55\%$55% of samples are solvable without visual or temporal cues, and after filtering these shortcuts, the remaining video-native challenges expose a substantial capability gap where state-of-the-art Video-LLMs perform only marginally above random, thus providing a distilled testbed to evaluate robust video understanding.

Key Contributions

• Video-Oasis is introduced as a diagnostic suite that audits video understanding benchmarks using visual-temporal decoupling tests, cross-model consensus, and human verification to isolate samples requiring genuine spatio-temporal reasoning.
• The audit finds that 55% of existing benchmark samples are solvable without visual input or temporal context, and after filtering these shortcuts, state-of-the-art models perform only marginally above random guessing.
• Using the distilled video-native challenges as a testbed, the work identifies temporal grounding and adaptive reasoning as key drivers of robust understanding and surfaces the balance between supervised fine-tuning and reinforcement learning from video reasoning as an open question.

Introduction

The rise of video large language models (Video-LLMs) has pushed evaluation beyond narrow tasks toward integrated perception and reasoning, but it remains difficult to determine whether benchmark gains reflect genuine visual understanding, linguistic shortcuts, or external knowledge. Existing benchmarks often contain samples that can be solved without visual or temporal evidence, and prior auditing efforts have addressed isolated issues such as temporal shortcuts or perception-heavy design without jointly examining visual dependency, temporal dependency, and annotation quality across diverse benchmarks. The authors introduce Video-Oasis, a diagnostic suite that systematically decouples visual and temporal cues to identify shortcut-solvable samples and distill video-native challenges, revealing that over half of benchmark samples lack true video dependencies and that state-of-the-art models perform only marginally above chance on the remaining rigorous tasks.

Dataset

• Dataset composition and sources Video-Oasis is a distilled evaluation set built from 14 existing public video QA benchmarks. The original collection contains 24,416 multiple-choice QA pairs linked to 4,938 unique videos. After filtering, the dataset retains 11,033 QA pairs, reducing the evaluation volume by 55%.

• Filtering rules The authors apply a set of diagnostic tests (Video-Oasis) that remove samples solvable without genuine spatio-temporal reasoning—for example, questions answerable from a single frame, through linguistic or auditory cues alone, or those with ambiguous grounding. Only QA pairs that require visual and temporal dependencies survive.

• Challenge categories The remaining samples are organized into five video-native challenge types: Fine-Grained Perception, Spatial World Understanding, Temporal Dynamics & Tracking, Causality & Logical Reasoning, and Global Narrative. Each QA pair is assigned a primary category by an ensemble of five proprietary LLMs; a label is accepted when at least three models agree, and the 122 cases without consensus are manually inspected and labeled.

• Usage The dataset is used solely for evaluation, not for training. It provides a more efficient and rigorous benchmark that specifically targets spatio-temporal video understanding, avoiding shortcuts that earlier benchmarks allowed.

• Processing details Beyond the filtering step, no additional cropping or preprocessing is applied to the videos. The category metadata is constructed through the LLM ensemble labeling pipeline, with manual verification for ambiguous instances.

Method

The authors introduce Video-Oasis, a diagnostic suite designed to verify whether video benchmarks satisfy the essential criteria for genuine video understanding. The framework systematically examines visual, temporal, and annotation dependencies to filter out shortcut-solvable samples.

As shown in the figure below, the diagnostic suite operates through three primary criteria. First, to test visual dependency, the authors replace the original video with inputs that remove or abstract raw visual evidence. This includes a Blind test providing only the question and options, an Audio test using transcribed audio tracks, and a Summary test using concatenated captions. If a model succeeds here, the task likely relies on linguistic bias rather than visual perception. Second, to verify temporal dependency, the authors employ a Center-Frame test using only the middle frame, a Frame Shuffling test to disrupt chronological order, and a Bag-of-Frames test using a frozen CLIP-based encoder for similarity matching without temporal modeling. Success in these tests indicates the task may not require true temporal reasoning. Third, to ensure annotation reliability, the authors apply Consistency, Redundancy, and Sensitivity checks to flag ambiguous samples for manual inspection.

The manual verification process addresses specific annotation issues identified by the diagnostic checks. For instance, consistency and redundancy checks help identify incorrect temporal labels or ambiguous subjects that prevent reliable anchoring of the answer to the video. Similarly, cases initially filtered by frame shuffling are manually reviewed to restore samples that genuinely require temporal ordering, such as reasoning about events following a specific action.

Following the application of Video-Oasis, the authors distill the remaining samples into video-native challenges. They aggregate source-benchmark metadata and prompt an LLM to derive candidate challenge clusters, which are then consolidated into five unified categories: Fine-Grained Perception, Spatial World Understanding, Temporal Dynamics & Tracking, Causality & Logical Reasoning, and Global Narrative. To assign each QA pair to a primary category, an ensemble of five proprietary LLMs is employed, with a label accepted upon majority agreement. This process filters out shortcut-driven samples and highlights inherent video-native challenges that demand strict spatio-temporal dependencies.

Finally, the authors utilize this distilled set to explore algorithmic designs, specifically focusing on adaptive reasoning. Using Qwen3-VL as a base model, they compare instruction-following modes, thinking modes, and adaptive thinking via VideoAuto-R1. The results suggest that reasoning depth should be adjusted dynamically based on the question rather than fixed, as an oracle ensemble baseline that optimally selects between thinking and non-thinking states nearly closes the performance gap with frontier models.

Experiment

The study first audits 14 video benchmarks using diagnostic tests that remove or disrupt visual and temporal evidence, revealing that a large majority of samples can be solved via shortcuts rather than genuine spatiotemporal reasoning. After filtering these shortcut-solvable instances, a distilled evaluation set is created, on which most current video-language models perform near chance, with global narrative understanding as a key bottleneck. Ablation studies further show that precise temporal grounding and adaptive reasoning strategies yield significant gains on the distilled challenges, and that supervised fine-tuning and reinforcement learning with verifiable rewards offer complementary benefits for strict video understanding.

Video-Oasis audits 14 benchmarks, substantially more than prior studies that examined 2 to 9. It is the only work that provides multiple tests for all three diagnostic axes—visual, temporal, and ambiguity—and the only one that combines cross-model consensus with manual verification. Prior audits either lacked visual diagnostics entirely (EgoTempo) or offered only a single temporal test (Cambrian-S, Apollo), while Video-Oasis includes multiple tests for each axis. Manual verification was absent in EgoTempo and Cambrian-S, and cross-model consensus was not used in any previous study, making Video-Oasis the first to employ both.

When visual evidence is removed or temporal order is disrupted via diagnostic tests, video-language models still achieve accuracies between 33% and 36%, well above the 25.6% random baseline. This indicates that many benchmark samples contain shortcuts, allowing models to answer correctly without relying on intended visual or temporal dependencies. Under audio-only, summary, center-frame, and frame-shuffling tests, Eagle2.5, Qwen2.5-VL, and Qwen3-VL reach 35.6%, 33.5%, and 36.2% accuracy, respectively, far exceeding the 25.6% chance level. The observed 30–50% accuracy range across all models on disrupted inputs reveals that a substantial portion of benchmark instances can be solved without genuine spatiotemporal reasoning.

Shortcut-solvable samples are pervasive across all task groups, with an average of 92.7% of instances exhibiting shortcut behavior when at least one diagnostic model agrees. Even under the strictest consensus of three models, a majority of spatial, temporal, and general samples remain vulnerable to shortcuts, while reasoning tasks show a lower but still substantial 44.6% shortcut ratio. Under a relaxed consensus threshold (c ≥ 1), over 94% of spatial, temporal, and general samples are shortcut-solvable, and reasoning tasks still reach 85.8%. At the strictest consensus (c = 3), shortcut ratios remain above 50% for spatial (58.8%), temporal (54.4%), and general (63.0%) benchmarks, while reasoning drops to 44.6%, indicating relatively greater robustness.

Manual refinement targeted three types of ambiguity: consistency, redundancy, and sensitivity. The sensitivity check, which corrects potential false positives from frame shuffling, required the most extensive refinement, with a substantially larger number of samples both flagged and corrected compared to the other two checks. Consistency and redundancy checks addressed smaller but meaningful sets of unreliable annotations. Sensitivity checks accounted for more refined samples than consistency and redundancy checks combined. Consistency and redundancy checks each refined around 200 samples, indicating a non-trivial level of annotation unreliability.

Summary, Center-Frame, and Frame Shuffling tests together uncover 65% of all unique shortcuts, forming a compact and effective protocol for benchmark refinement. Frame Shuffling alone identifies the most distinct shortcuts, while Audio and Blind tests contribute the fewest unique detections due to substantial overlap with other diagnostics. Summary, Center-Frame, and Frame Shuffling tests account for 65% of unique shortcuts. Frame Shuffling yields the largest number of distinct shortcuts, whereas Audio and Blind tests are the least discriminative.

Video-Oasis audits 14 benchmarks with multiple diagnostic tests per axis, cross-model consensus, and manual verification. Models achieve far above random accuracy on disrupted inputs, revealing pervasive shortcuts across tasks. Manual refinement primarily corrects sensitivity issues, and a compact set of tests uncovers most unique shortcuts.