One-sentence Summary

University of Hong Kong, Nanjing University, et al. propose OSReward, a realistic cross-platform benchmark for evaluating vision-language model judges of computer-using agent trajectories, find a systematic leniency bias even in state-of-the-art judges, and release OS-Shepherd-100K to train open OS-Shepherd (9B and 35B) reward models that match commercial performance at $30\text{--}60\times$30–60× lower cost.

Key Contributions

• The paper introduces OSReward, a cross-platform benchmark of computer-using agent trajectories with ground-truth verdicts from multi-stage human annotation, plus a hard subset (OSReward-Hard) and a multi-score variant (OSReward-Multi).
• A comprehensive evaluation of vision-language model judges reveals a systematic leniency bias that mislabels incomplete trajectories as successes, causing performance on OSReward-Hard to drop near chance.
• The work constructs OS-Shepherd-100K, an open reasoning-annotated corpus, and trains OS-Shepherd (9B and 35B), open reward models that match commercial judges at 30–60× lower cost and stay de-biased on unseen benchmarks.

Introduction

Computer-using agents that operate web, mobile, and desktop interfaces require scalable trajectory-level verification to drive evaluation, data curation, and reinforcement learning. Human-written verifiers cover only a handful of curated tasks and cannot handle static corpora or previously collected trajectories, while human annotation cannot keep pace with the volume. The de facto practice falls to a VLM acting as a judge, but its reliability remains largely untested: prior studies are confined to single platforms, reuse off-the-shelf benchmarks with imperfect ground truth, and rely on short, narrow evidence. The authors address this gap by building OSReward, the first cross-platform benchmark with human-gold trajectories collected from fresh environments across web, mobile, Ubuntu, and Windows, and then releasing OS-Shepherd, an open dataset of 100K reasoning-annotated judgments and two open reward models that match expensive frontier judges at a training-scale budget.

Dataset

The authors build two datasets: a gold benchmark (OSReward) for evaluating CUA judges and a large-scale training corpus (OS-Shepherd-100K) for training the OS-Shepherd reward model. Both are collected on their own cross-platform infrastructure, with trajectories consisting of a task instruction, a sequence of steps (screenshots, agent thoughts, and actions), and a verdict.

OSReward benchmark (gold labels)

• Sources and composition
  • 1019 trajectories spanning Web, Windows, Ubuntu, and Mobile environments.
  • Tasks are written by human annotators after exploring the live environments; ~800 of ~1500 candidates survive a peer screening that removes ambiguous or ungrounded instructions.
  • Each verified instruction is executed by one to three agents from diverse backbones (Claude, Gemini, Kimi, Qwen families), mixing GUI-only and GUI+CLI action spaces to reflect real-world agent diversity.
  • Runs with severe collection problems (anti-bot blocks, network failures, frozen executions) are pre-filtered before human annotation.
• Human annotation and labeling
  • Every trajectory is labelled independently by three annotators; unanimous labels are final, split cases go to a meta-review by senior reviewers.
  • Annotation cost: roughly 800 human hours.
  • Annotators read the full multimodal context and apply a strict rule: an answer not obtained/verified through the environment is a FAIL even if correct.
  • Failed trajectories are tagged with error categories (reasoning, action, perception, memory).
  • Successful trajectories carry two sub-labels on OSReward-Multi: alignment (2 levels) and efficiency (3 levels).
• Subsets
  • OSReward (full set): 1019 trajectories, balanced across platforms, with a 43% SUCCESS / 57% FAIL split. Reasoning-and-planning errors dominate failures (86% of failed runs).
  • OSReward-Hard: 284 trajectories, mostly drawn from cases where annotators initially disagreed. Re-verified with extra review. The split is 30% SUCCESS / 70% FAIL, designed to expose false-success vulnerabilities in judges.
  • OSReward-Multi: the 440 successful trajectories from the full set, with alignment and efficiency ratings; failure trajectories are excluded.
• Usage
  • The benchmark evaluates judge models; OSReward-Hard isolates difficult cases, and OSReward-Multi provides a finer-grained quality axis for successful runs.

OS-Shepherd-100K training corpus (auto-labeled)

• Construction and scale
  • 100K trajectory-judgment pairs built for cost-effective training. Labels are automatically assigned by an ensemble of strong VLM judges, keeping only trajectories where diverse judges reach high agreement.
  • The ensemble varies judge models and screenshot settings to reduce herd bias; ambiguous trajectories are discarded, and no forced majority label enters the corpus.
  • Each retained sample carries the judge’s reasoning, not just a binary verdict.
• Sources and composition
  • Covers Web, Windows, MacOS, Ubuntu, and Android.
  • Trajectories come from five model families (Claude, Gemini, GPT, Kimi, Qwen) under multiple harnesses, action spaces, and step budgets, plus open-source agent stacks.
  • About 46% of Ubuntu data interleaves GUI actions with command-line steps.
  • The corpus is disjoint from the OSReward benchmark trajectories.
  • To reach scale, a portion of instructions are synthesized: ~25% of combined Ubuntu and Windows instructions, ~10% of web instructions.
• Filtering and label provenance
  • An agreement filter retains roughly 85% of judged trajectories; only near-unanimous outputs from the strongest judges are kept.
  • The final label is derived from the ensemble; the retained reasoning text defaults to one strong judge’s output, never contradicting the label, with occasional exposure to other judges’ reasoning for generalization.
  • Success/failure mix is balanced, and negatives are over-collected from real failures, not environment artifacts.
• Usage
  • Used solely to train the OS-Shepherd reward model; the training recipes are not disclosed here beyond the dataset’s role.

Data infrastructure and processing

• All environments (Web, Windows, Ubuntu, Mobile) are richly initialized with everyday applications, real files, seeded databases, and distractor content to produce realistic, judgeable trajectories. Web tasks run on live websites.
• Collection is uniform across platforms, yielding a consistent trajectory format (instruction, step-by-step screenshots with agent thoughts and actions) that both the benchmark and training corpus share.
• No cropping strategy is applied; the full screen content is preserved for multimodal judging.

Method

The authors leverage a comprehensive pipeline to construct OSReward, a corpus of realistic, cross-platform computer use agent trajectories paired with trustworthy gold verdicts. The data collection infrastructure spans Web, Windows, Ubuntu, and Mobile environments, each initialized with everyday applications, real files, and seeded databases to ensure realistic states. Annotators explore these environments to draft grounded task instructions, which undergo a rigorous peer cross-check to eliminate ambiguous or unanswerable prompts. Surviving instructions are then executed by diverse agent frameworks driven by mainstream model backbones, such as Claude, Gemini, Kimi, and Qwen, generating raw trajectories that capture varied action idioms and failure modes.

To produce the gold standard labels, the raw trajectories pass through a multi-stage annotation pipeline. An automatic pre-filter first discards runs suffering from severe collection issues like network failures or anti-bot blocks. The remaining trajectories are independently labeled by three human annotators who review the full multimodal context, including every screenshot, thought, and action. A strict standard is applied where an unverified correct answer is marked as a failure. When annotators unanimously agree, the verdict is final. Disagreements escalate to a meta-review conducted by senior reviewers. Successful trajectories are scored on alignment and efficiency axes to form the OSReward-Multi set, while failures are categorized by error type. Hard cases that split annotators undergo further re-verification to create the OSReward-Hard set.

To address the need for a scalable reward model, the authors develop OS-Shepherd, trained on the OS-Shepherd-100K corpus. Since human annotation is too costly for training-scale data, the authors employ an ensemble of strong vision-language model judges. The pipeline begins with filtering self-collected instructions and executing them via agents, then joins these with selected open-source trajectories. Each trajectory is scored by multiple judge instances under varying input screenshot settings. To mitigate judge bias and herding, the authors retain only trajectories where diverse strong judges independently reach a near-unanimous verdict, discarding ambiguous cases. This cross-model agreement filter ensures the training set contains reliable labels and diverse reasoning traces, resulting in the final OS-Shepherd-100K dataset.

The OS-Shepherd model is trained in two stages starting from a Qwen3.5 base. In the supervised fine-tuning stage, the base vision-language model is fine-tuned on the agreement-filtered OS-Shepherd-100K corpus. This step significantly corrects the base model's inherent leniency. Following this, a reinforcement learning stage leverages Group Relative Policy Optimization to target false successes, the most harmful errors for a reward signal. By mining these cases and applying RL, the model further refines its ability to distinguish correct behavior from incorrect trajectories.

Experiment

A benchmark of 27 VLMs as judges for computer-using agents reveals that only frontier models approach usable accuracy, but all suffer a dominant leniency bias where they over-accept failed tasks as successes because verdicts rely heavily on the agent's textual claims rather than visual evidence. Analyses show that removing the agent's text history causes large drops in accuracy while altering visual inputs barely matters, and neither extra reasoning nor ensembling substantially improves reliability. The study introduces OS-Shepherd, a small open reward model that matches commercial judges' accuracy, resists the leniency bias, and generalizes across independent benchmarks at a fraction of the cost.

On the OSReward benchmark, only frontier closed-source judges approach the roughly 90% accuracy needed for a training-time reward, while open-weight counterparts trail across the field, with the largest open models narrowing the gap to within a few points. The OSReward-Hard challenge set reveals that this aggregate accuracy is fragile: every judge's performance drops sharply, the best falling to the level of a trivial majority-class predictor, and leniency widens dramatically as many judges accept almost all deceptive failures. On this harder footing, only two frontier models remain both relatively accurate and balanced near 70% recall on each side, while a small open model trained with de-biasing (OS-Shepherd) transfers this resistance across unseen benchmarks, making it the best-calibrated open judge. The top closed-source judges achieve accuracy just below 90% on OSReward, but their ordering shifts when evaluated by balanced accuracy instead of raw accuracy. Open-weight judges consistently underperform closed-source ones, though the largest open models approach within a few percentage points of the frontier. Switching to the harder OSReward-Hard set causes every judge's accuracy to collapse by up to 43 points, with the best judge scoring no better than a constant predictor on the skewed success/failure mix. Leniency bias widens on the hard set: several models recall almost zero failures, while the most lenient accept nearly all hard failures, inflating the spread of failure recall across judges. Only Claude-Opus-4-8 and GPT-5.5 sustain roughly 70% recall on both successes and failures in the hard set, while other judges near the diagonal achieve balance only at the expense of lower accuracy on both axes. OS-Shepherd models, trained for leniency resistance, transfer this de-biasing out of distribution, outperforming all general open judges on independently built benchmarks and narrowing the gap to human-written verifiers.

On the OSReward-Multi benchmark, quality grading proves far weaker than binary outcome judging, with even the best judge reaching only low-60s AUC. Judges consistently rank alignment and efficiency levels better than their emitted scores indicate, revealing a miscalibration gap that is most severe on alignment, where models tend to default to the top rating. The open-source OS-Shepherd models inherit this weakness, though the 35B variant partially closes the calibration gap while the 9B remains near a constant-level baseline. The top judge achieves an overall multi-axis AUC above 66%, but its macro-recall stays under 59%, illustrating the gap between ranking ability and score calibration. Alignment grading suffers most from miscalibration: some judges attain alignment AUC over 71%, yet overall macro-recall remains low, reflecting a tendency to output the highest rating regardless of trajectory quality. OS-Shepherd-35B improves macro-recall to 47.7% compared to the 9B's 44.1%, partially reclaiming the calibration gap, while both trail frontier models in AUC.

The OS-Shepherd-100K judge-instance pool contains over 320k verdicts across five platforms, with web interactions forming the largest share. Ubuntu trajectories contribute about one-fifth of the pool, and nearly half of those combine GUI and command-line actions. After filtering for high-agreement ensembles, the retained training samples concentrate on desktop and web, while mobile keeps only a minor presence. Web instances dominate the pool at 37%, while desktop platforms (Windows, macOS, Ubuntu) collectively supply over half of all judge instances. Within the Ubuntu subset, 46% of trajectories interleave GUI and command-line interactions, reflecting deliberate collection of combined GUI-CLI behavior.

Fine-tuning turns a lenient base model into a balanced reward model that catches far more hard failures while staying near the diagonal. The 9B version moves from the bottom third of the field into the commercial range on the full set, at a fraction of the cost, and scaling to 35B yields only marginal further gains, indicating that the recipe matters more than parameter count. Training OS-Shepherd-9B lifts its full-set balanced accuracy from 79.4 to 86.3 and hard-set balanced accuracy from 55.9 to 61.9. On hard instances, the tuned 9B captures 57.6% of false successes, a leap from 14.1% for its base model, while keeping success recall and failure recall nearly equal. The 35B model, with four times the parameters, adds only 2.4 percentage points of hard-set balanced accuracy over the 9B and matches it on the full set. OS-Shepherd-9B costs about $1.36 to judge the full benchmark, roughly a third less than the nearest commercial judge while leading on the hard subset. The learned de-biasing transfers across independently built CUA benchmarks, where the 9B catches far more true failures than lenient general judges.

Frontier closed-source judges approach 90% binary accuracy on standard computer-use reward benchmarks but collapse on a hard challenge set with deceptive failures, while OS-Shepherd, a low-cost fine-tuned open model, learns balanced leniency resistance and transfers de-biasing across unseen benchmarks. Multi-axis quality grading reveals severe miscalibration, especially for alignment, which OS-Shepherd partially mitigates. Training on over 320k verdicts from diverse desktop and web platforms and retaining high-agreement samples turns a lenient base model into a cost-effective 9B judge that matches commercial models on standard tests and leads on hard cases.