One-sentence Summary

The authors propose UI-Venus-2, a large language model for GUI agents that achieves state-of-the-art results on MobileWorld (GUI-only success rate over 117 tasks with 50 steps), WebVoyager (refreshed 595-task split), Odysseys (average rubric score over 200 tasks), VenusBench-CAPTCHA (micro Pass@1 over all 219 examples), and VenusBench-GD (English-instruction micro-average accuracy), outperforming strong baselines in standalone end-to-end evaluations.

Key Contributions

• UI-Venus-2 is a GUI agent that jointly scales multilingual environments, desktop computer-use capabilities, and fine-grained verification mechanisms to achieve state-of-the-art performance.
• The model, available in 27B and 9B sizes, sets new state-of-the-art results on GUI-agent benchmarks including MobileWorld, WebVoyager, Odysseys, VenusBench-CAPTCHA, and VenusBench-GD.
• UI-Venus-2 generalizes across mobile, web, and desktop platforms, supporting realistic everyday use on diverse operating systems.

Introduction

Multimodal large language models have enabled GUI agents that can perceive screenshots and simulate human actions like clicking and typing, automating apps without relying on structured APIs. However, prior agents struggle outside narrow benchmarks because they are trained on limited app sets, rely on queries that may not reflect real app functionality, and often use reward verifiers that mistake partial progress for success. The authors introduce UI-Venus-2, a general-purpose foundation agent that jointly scales the environment pool across multilingual mobile apps, desktop OS, and web platforms; constructs large volumes of executable tasks grounded in actual application capabilities; and employs a robust two-level verification system that evaluates true task completion via visual keypoints and multi-model voting.

Dataset

The authors construct a multimodal GUI interaction dataset using a closed‑loop pipeline that generates, executes, and verifies trajectories across four domains: web navigation, computer use, synthetic GUI grounding, and synthetic CAPTCHA. Data is filtered and annotated through trace‑level and step‑level verification before being used for supervised fine‑tuning (SFT) and reinforcement learning (RL).

Dataset composition and sources

• Web Navigation

  • Environment pool: Over 4,000 domains from 19 categories, selected by combining public browser‑agent benchmarks with the Tranco ranking and scoring on dynamicity, interactivity, and visual quality via an automated VLM (Kimi 2.6).
  • Seed tasks: 45,000 tasks adopted from InSTA‑150k‑v3, prioritized by rich success‑criteria annotations.
  • Collection: Trajectories recorded in real Chrome sessions through a 15‑action Playwright interface.
  • Post‑processing: Step‑level sanitization removes redundant waits, opposite‑direction scrolling, and cyclic action patterns; recoverable errors in successful trajectories are kept to enrich failure‑mode coverage in the capability catalog.

• Computer Use

  • Tasks: Each task is serialized as a TaskSpec containing a controlled desktop snapshot, setup operations, required files/services, and an outcome evaluator.
  • Pre‑execution checks: Materialization plans provision the workspace; fixture fingerprints deduplicate environments; preflight checks validate initial state integrity.
  • Long‑horizon handling: Hierarchical segmentation splits tasks into subgoals with independent artifacts and completion checks; exit states seed recollection for subsequent segments.
  • Collection: Real desktop interactions are rolled out, and both successes and failures feed back into the capability catalog.

• Synthetic GUI Grounding

  • Generation: Executable HTML/CSS/JS interfaces are synthesized from natural‑language scenarios, optionally conditioned on personas or reference screenshots.
  • Target identification: Candidate targets are validated via a nine‑point hit test (visibility, viewport clipping, occlusion, etc.) and text‑node refinement.
  • Annotations: Accepted targets carry coordinates, visible text, ARIA attributes, and local context; infeasible instructions serve as hard negatives.
  • Usage: Exported directly for grounding‑specific SFT and RL, bypassing the general task‑construction stage.

• Synthetic CAPTCHA

  • Types: 70 CAPTCHA categories, each driven by a rule engine that stores latent state defining the answer, target geometry, valid actions, and solution trace.
  • Rendering: A renderer composites puzzles into mobile panels or webpage contexts, mapping coordinates to the final canvas; puzzle validity is verified before rendering.
  • Records: Include rendered observations, normalized geometry, reasoning supervision, and executable action sequences.
  • Usage: Exported for CAPTCHA‑specialized SFT and RL, sharing export conventions with other domains.

How the data is processed and used

• Trace‑level verification (Semantic Guided Verification)

  • A VLM‑as‑Judge inspects full trajectories to determine if the task objective was semantically satisfied, producing keypoint‑based evidence and a final classification: completed, partial, infeasible, or failed.
  • Completed trajectories flow into a high‑quality pool for training; partial trajectories can be repaired; infeasible cases refine task feasibility rules; failed samples help separate agent capability from task design issues.
  • This verification is a curation mechanism rather than a direct training label; newly discovered functionalities feed back into the capability catalog.

• Sample‑level verification

  • Each interaction step is assessed a priori (before execution outcome) as correct, exploratory, ineffective, or incorrect.
  • Step judgments are aggregated into a three‑tier trajectory label that distinguishes generic operations (app launch, scroll) from task‑specific operations (entering content, selecting targets). A trajectory receives partial credit only if it contains at least one valid task‑specific operation.
  • These fine‑grained signals provide a more reliable reward signal for RL and help filter training data.

• Training usage

  • Web and computer‑use trajectories, after verification, are used for SFT and RL to teach the agent real‑environment interaction.
  • Synthetic grounding and CAPTCHA records provide dense supervision for their respective modalities and are used directly for SFT and RL.
  • All domain outputs continuously update the shared capability catalog, guiding subsequent rounds of task generation.

Method

The authors present UI-Venus-2, a general-purpose foundation GUI agent that combines visual perception and advanced reasoning with reinforcement learning-driven interaction capabilities. Given a natural language instruction, the model operates through a unified reasoning and action framework. It observes rendered interface images, interprets the visual context, translates high-level user intent into executable GUI actions, and continuously adapts its decisions based on environmental feedback. The training pipeline integrates multimodal mid-training, offline reinforcement learning, and multi-teacher on-policy distillation. The model is initialized from strong open-source multimodal foundation models to provide robust priors in visual understanding and instruction following.

To support this interaction paradigm, the authors design a comprehensive data generation and verification framework.

Refer to the framework diagram.

The data generation pipeline operates as a closed-loop iterative system. It begins with capability catalog construction, which distills application knowledge from heterogeneous sources into a structured registry. Guided by this catalog, the task construction module synthesizes diverse task types, formalizing each as an executable contract that binds natural language instructions to application domains and expected outcomes. A validity gate enforces strict constraints to reject ambiguous or unsupported tasks. During trajectory collection, the executor interacts with the environment through a screenshot and action loop. Both successful and failed executions yield structured feedback, which updates the capability catalog and informs coverage-aware sampling for subsequent rounds. This process spans mobile, browser, and computer environments, ensuring diverse and scalable interaction data.

The first training stage involves multimodal mid-training on a large-scale mixture of synthetic and interaction-oriented data. Mobile, web, and operating system navigation tasks constitute the dominant component. The authors simulate a broad range of interactive environments to collect executable trajectories. Task queries are constructed through a complementary combination of curated seeds to balance realism and linguistic diversity. The resulting trajectories undergo a collaborative verification process combining human inspection and automated assessment to filter low-quality interactions.

Building upon the mid-trained models, the second stage applies offline reinforcement learning with task-specific supervision. For mobile, operating system, and web environments, the authors construct large-scale step-level reinforcement learning trajectories. This provides supervision at individual interaction steps to optimize state-aware action selection and multi-step navigation. For CAPTCHA and grounding tasks, a programmatic synthesis framework embeds verified instances into realistic interfaces, enabling dense and scalable reinforcement learning supervision that ensures robust target identification and precise spatial localization.

The final stage employs multi-teacher on-policy distillation to consolidate capabilities acquired from heterogeneous task distributions. To mitigate interference from directly merging independently trained experts, the authors adopt an on-policy distillation paradigm where teachers score trajectories sampled by the student. Recognizing that vanilla on-policy distillation applies uniform token-level supervision, which is suboptimal for GUI agents, the authors introduce structured action-aware distillation. Since the action is the sole interface for environment interaction and exhibits internal structural dependency, the token-level distillation signal is conditioned on the correctness of the student action. If the complete action is correct, the distillation signal is suppressed. If the action type is correct but parameters are flawed, supervision over the action span is strengthened. If the action type is incorrect, the type tokens are emphasized while downstream parameters are masked.

Furthermore, the authors implement teacher-side action-type conditioning. During training, the routed teacher scores the response sampled by the student. To provide precise supervision on structured actions, a hint regarding the correct action type is appended to the teacher prompt. This hint is never included in the student prompt and is unavailable at inference time. The token-level distillation advantage is computed as:

$$\widehat {A} _ {t} ^ {\mathrm{hint}} = \mathrm{sg} [ \log \pi_ {T _ {d}} (y _ {t} | \mathcal {P} _ {T} (x, z ^ {*}), y _ {< t}) - \log \pi_ {\theta} (y _ {t} | \mathcal {P} _ {S} (x), y _ {< t}) ]$$Athint​=sg[logπTd​​(yt​∣PT​(x,z∗),y<t​)−logπθ​(yt​∣PS​(x),y<t​)]

where $\widehat {A} _ {t} ^ {\mathrm{hint}}$Athint​ is the distillation advantage for the t-th student-sampled token $y _ {t}$yt​, and $y _ {< t}$y<t​ denotes its preceding context. $T _ {d}$Td​ is the frozen teacher for domain $d$d, and $\pi _ {\theta}$πθ​ is the student policy. The original GUI input is denoted by $x$x, and $z ^ {*}$z∗ is the correct action type. $\mathcal {P} _ {T} (x, z ^ {*})$PT​(x,z∗) represents the teacher prompt augmented with the hint, whereas $\mathcal {P} _ {S} (x)$PS​(x) is the unchanged student prompt. The operator $\mathrm{sg}[\cdot]$sg[⋅] stops gradients through the advantage computation. This design ensures that the distillation signal is effectively concentrated on critical executable behaviors, yielding robust fusion results across diverse GUI domains.

Experiment

The experiments evaluate UI-Venus-2 models across a comprehensive set of GUI agent benchmarks covering mobile use, computer use, web navigation, GUI grounding, CAPTCHA solving, and safety. Compared to general vision-language models and specialized GUI agents, UI-Venus-2 consistently achieves top-tier or state-of-the-art results, with particular strengths in mobile, web, and CAPTCHA tasks, while also demonstrating competitive grounding and significantly improved safety against both explicit attacks and benign-looking harmful instructions. The 9B variant often rivals much larger models, and the 27B variant further extends performance, indicating robust generalization and effective scaling across diverse real-world scenarios.

General-purpose vision-language models show uneven performance across mobile GUI benchmarks, with no single model dominating all tasks. Seed-2.0-Pro achieves the highest scores on MobileGym, KnowUBench, and MemGUI, while Kimi-K3 leads on MobileWorld and Qwen3.6-27B leads on AndroidWorld. Larger model variants consistently outperform their smaller counterparts, as seen in the Qwen and Kimi families. Seed-2.0-Pro reaches 52.0 on MobileGym, 51.6 on KnowUBench, and 65.6 on MemGUI, but scores only 20.1 on VenusBench-Mobile. On MobileWorld, Kimi-K3 attains 74.4, surpassing Seed-2.0-Pro (63.2) and other models. Qwen3.6-27B outperforms Qwen3.5-9B on every benchmark where both are reported, with gains of 15.6 points on MobileGym and 12.7 points on VenusBench-Mobile. The 100-step setting on MobileWorld raises Qwen3.6-27B from 36.8 to 41.9, indicating that additional steps can benefit some models.

UI-Venus-2 models establish new state-of-the-art on live-web navigation benchmarks, with the 27B variant leading across WebVoyager, Online-Mind2Web, and REAL. The smaller 9B model also performs strongly, often surpassing larger general-purpose systems and rivaling top-tier agents. UI-Venus-2-27B achieves 93.4% on WebVoyager, outdoing the strongest GUI-based baseline by4.1 points. UI-Venus-2-9B reaches90.8% on WebVoyager, remaining competitive with GPT-5 (SoM) at a much smaller scale. On Online-Mind2Web, UI-Venus-2-9B scores74.0%, exceeding the larger Fara1.5-27B. UI-Venus-2-27B and -9B surpass the previous best on REAL by5.8 and2.5 points, respectively.

UI-Venus-2-27B achieves top accuracy among GUI-specific models on VenusBench-GD and OSWorld-G-R, and ranks second on ScreenSpot-Pro and UI-Vision, consistently outperforming strong general vision-language models and closed-source baselines. The smaller 9B variant significantly narrows the gap to its larger counterpart, delivering robust grounding across diverse interfaces with substantially fewer parameters. UI-Venus-2-27B improves over its predecessor by 5.1 points on VenusBench-GD and by12.2 points on UI-Vision, demonstrating consistent gains on both comprehensive and visually challenging grounding tasks. The 9B model nearly matches the 27B model on ScreenSpot-Pro and OSWorld-G-R, showing that precise GUI grounding can be achieved at a smaller scale.

UI-Venus-2 models substantially outperform general-purpose VLMs on CAPTCHAs that require translating recognition into executable actions and on diverse reasoning mixtures, with the largest gains on tasks combining multiple primitives. Performance remains competitive on spatial and heterogeneous benchmarks, where the best general models nearly match the 27B variant, indicating ongoing challenges in perspective taking and multi-step transformations. The 9B model preserves most of the capability, often surpassing the strongest general baselines by wide margins. On VenusBench-CAPTCHA, UI-Venus-2-27B and -9B achieve 79.9% and 78.1% Pass@1, compared to 53.0% for the best general VLM, showing effective translation of CAPTCHA understanding into valid interface actions. Across the category-balanced MCA-Bench subset, the 27B model reaches 79.6% and the 9B reaches 75.7%, while Qwen3.6-27B attains 51.7%, indicating broad generalization beyond common formats. The clearest separation ap pears on NextGen-CAPTCHAs, where UI-Venus-2-27B achieves 54.5% versus Seed-2.0-Pro's 20.4%, and the 9B model more than doubles the best general baseline. On Spatial-CAPTCHA-Bench, the 27B model leads at 48.6%, only 4.8 points above Seed-2.0-Pro, and the 9B variant trails by just 1.0 point, showing that spatial transformations remain a bottleneck for all models. Open CaptchaWorld yields the tightest race : UI-Venus-2-27B at 56.3% is merely 0.7 points ahead of Seed-2.0-Pro, and the 9B model (50.7%) still outperforms Kimi-K2.6 by 2.9 points, demonstrating broad coverage with limited headroom.

A safety evaluation on OSHarm and OSBlind benchmarks shows that existing general-purpose and GUI-specific models are highly susceptible to OSBlind attacks, with attack success rates exceeding 79%. In contrast, UI-Venus-2 models cut the OSBlind attack success rate to approximately 48%, nearly halving the risk, and also achieve the lowest rates on OSHarm, outperforming all baselines. These results demonstrate that UI-Venus-2 effectively resists both explicit threats and safety blind spots during benign execution. UI-Venus-2 models attain the lowest attack success rates on OSHarm, surpassing the strongest baseline by a notable margin. On OSBlind, UI-Venus-2 reduces the attack success rate from over 79% (baselines) to around 48%, nearly halving the vulnerability.

The evaluation spans mobile GUI benchmarks, live-web navigation, grounding, CAPTCHA reasoning, and safety attacks. General-purpose vision-language models show uneven performance across mobile tasks with no single dominant model, while UI-Venus-2 establishes new state-of-the-art on web navigation and on CAPTCHAs that require translating recognition into executable actions, and its smaller 9B variant frequently outperforms larger general-purpose systems. Grounding results are competitive with strong general models, though spatial reasoning remains a bottleneck for all systems. In safety tests, UI-Venus-2 nearly halves the OSBlind attack success rate compared to existing models, demonstrating robust resistance to both explicit threats and safety blind spots.