One-sentence Summary

Alibaba Group presents Qwen-UI-Agent, a real-world centric foundation GUI agent that unifies mobile, computer-use, web, and DeepSearch environments with a unified action space interleaving GUI and CLI operations, batched actions, and an AutoResearch-style data flywheel with online RL training across over 10,000 concurrent environments, achieving state-of-the-art mobile-use results including 82.1% on MobileWorld and 92.2% on MobileWorld-Real, alongside competitive computer- and browser-use performance against frontier models such as Opus 4.8, Gemini 3.1 Pro, and GPT-5.6 Sol.

Key Contributions

• Qwen-UI-Agent is a foundation GUI agent that unifies mobile, computer-use, web, and DeepSearch environments with a combined GUI and CLI action space generating batched actions in a single model turn.
• An AutoResearch-style data flywheel autonomously constructs tasks and environments, diagnoses failures, and plans iterative improvements, paired with an online RL training framework supporting trajectories over 100 turns across more than 10,000 concurrent environments.
• Qwen-UI-Agent achieves state-of-the-art mobile-use performance with 82.1% on MobileWorld, 92.2% on MobileWorld-Real, and 97.5% on AndroidDaily, while delivering competitive computer-use and browser-use scores against frontier models including Opus 4.8, Gemini 3.1 Pro, and GPT-5.6 Sol.

Introduction

Graphical user interfaces remain the dominant access layer for digital services, making capable GUI agents a promising path toward general-purpose task automation without requiring every application to expose a dedicated API. Although multimodal foundation models have driven rapid progress on simulated benchmarks, existing agents still struggle to transfer that performance to real-world devices, sustain long-horizon execution, or span workflows across mobile, web, and desktop environments. The authors introduce Qwen-UI-Agent, a foundation GUI agent designed to close this simulation-to-real gap by operating on physical mobile hardware, combining GUI and CLI actions within unified cross-platform workflows, scaling online reinforcement learning to trajectories exceeding one hundred steps, and automating much of the capability-development loop through an agent-driven data flywheel.

Dataset

The authors construct two distinct data resources: a training corpus generated by an agent-driven data flywheel, and a held-out evaluation benchmark called MobileWorld-Real.

Training data from the agent-driven flywheel

• Source and generation: Strong foundation models analyze domain knowledge and capabilities across mobile, desktop, and web. They produce an initial pool of tasks and environment contexts. Candidate trajectories are collected by executing these tasks.
• Filtering and quality control: Following the MAI-UI approach, multiple rounds of rejection sampling are applied. A VLM-based step-level judge inspects each trajectory and extracts maximal contiguous correct steps, the first step of each reflection or exploration phase, and recovery segments. Only these fine-grained segments are kept, recovering effective supervision from both successful and failed runs.
• Iterative refinement: After training an initial SFT policy, agent-driven failure analysis identifies model weaknesses and maps them to targeted optimization goals. New tasks, environments, and task-specific verifiers are synthesized to address these gaps, and accepted trajectories are added to the corpus, closing the loop.
• Usage: The filtered trajectories are aggregated into a unified SFT corpus to train a strong initial policy. Executable verifiers are built for tasks where reliable outcome checks are possible; these provide high-precision signals for online RL. Step-level judgment is used to scale process supervision for SFT, while executable verifiers supply outcome rewards for RL.

MobileWorld-Real evaluation benchmark

• Composition and scale: 409 end-to-end tasks covering 104 real Android apps, grouped into 7 daily-use domains: content consumption, life services, productivity, e-commerce, system settings, financial services, and social communication.
• Source and realism: Tasks are written by human contributors to reflect everyday needs. They run on live devices with real accounts, content, and networks, exposing agents to pop-ups, expired logins, permission requests, CAPTCHAs, and other real-world interruptions.
• Task characteristics: Nearly half of the tasks are labeled as hard, involving long-horizon execution, comparison and ranking, reasoning over changing information, deeply nested navigation, pop-up recovery, and cross-app coordination. Tasks may require the agent to ask for missing information or request user takeover.
• Processing and evaluation: Each run produces a full trajectory aligned with screenshots. AutoJudge, a trajectory-level evaluator, uses five independent VLM judges with majority voting to assign one of three outcomes: pass, failed, or env_error. Environment errors are reported separately and excluded from the success-rate denominator to reduce noise from live-device variability.
• Usage: All MobileWorld-Real tasks and collected trajectories are held out from training. The benchmark measures whether capabilities learned in controlled environments transfer to real devices, complementing sandbox evaluations.

Method

The authors design Qwen-UI-Agent as an integrated system for building advanced foundation GUI agents. The system comprises an environment infrastructure, an automated data flywheel, a unified training framework, and a harness layer.

A task is defined as $\tau = (I, \mathcal{E}_\tau)$τ=(I,Eτ​), where $I$I is the user instruction and $\mathcal{E}_\tau$Eτ​ is the set of digital environments. At decision step $t$t, the agent receives a multichannel observation $o_t = (o_t^{\text{GUI}}, o_t^{\text{CLI}}, o_t^{\text{API}})$ot​=(otGUI​,otCLI​,otAPI​) and predicts an intermediate reasoning output $r_t$rt​ and an executable action output $\mathbf{a}_t$at​:

where $h_t$ht​ denotes the preceding interaction history. The model can produce a single action or a batched action sequence $\mathbf{a}_t = (a_t^{(1)}, \dots, a_t^{(K_t)})$at​=(at(1)​,…,at(Kt​)​). The unified action space covers GUI interactions, CLI commands, and API calls. Refer to the framework diagram for an illustrative trajectory of proactive cross-platform task execution.

The environment infrastructure is organized around four core components to support scalable and realistic agent learning. First, simulated sandbox environments provide controllable settings for mobile, computer, browser, and deep search tasks. Second, a sim-to-real bridge extends agents to real devices with actual applications and network states. Third, a hybrid GUI and CLI action space interleaves graphical operations with direct command execution. Finally, a unified interface standardizes the thinking, action, and observation loop across heterogeneous environments. As shown in the figure below, this infrastructure spans from large-scale virtual sandboxes to real-world devices.

To address the instability of real devices, the authors build a real-device mobile runtime. As shown in the figure below, a health-aware scheduler continuously tracks the health of devices, applications, and networks, routing tasks to eligible runtimes and blacklisting unhealthy ones. Virtual displays allow a single physical phone to host multiple concurrent application sessions. A dedicated User Agent handles sensitive operations requiring explicit user approval. Furthermore, a VLM-based judge examines trajectories to distinguish task success, model failure, and environment failure, feeding confirmed environment issues back to the scheduler for recovery.

Continually improving the agent requires an iterative process. The authors introduce an agent-driven data flywheel that shifts human involvement to high-level oversight. As illustrated in the figure below, the flywheel consists of domain capability bootstrapping and an iterative refinement loop. During bootstrapping, strong foundation models analyze domain knowledge to generate initial tasks and environment contexts. In the iterative loop, the system evaluates performance, diagnoses failures, and synthesizes targeted tasks and verifiers to address specific weaknesses. Step-level VLM judges and trajectory-level verifiers provide scalable supervision for training.

The training framework combines supervised fine-tuning (SFT) with reinforcement learning. For SFT, the authors employ domain-conditioned expert training, merging specialized experts for mobile, desktop, and web into a unified model. They also use sliding-window training to efficiently process long trajectories. To correct recurring action errors such as confusable-element grounding or premature completion, they introduce Action RL. This stage uses targeted error-pattern data and an action-aware reward function:

where $F_t$Ft​ indicates format validity, $C_t$Ct​ and $Q_t$Qt​ measure action-type correctness and argument quality, and $S_t$St​ and $L_t$Lt​ penalize incorrect sensitive actions and repetitions. For long-horizon decision making, Online RL optimizes end-to-end task success using Group Relative Policy Optimization (GRPO) with a model-adaptive task curriculum that dynamically adjusts task difficulty based on the current policy capabilities.

Finally, the harness layer extends the core agent beyond isolated task execution. It connects signals from the user digital environment to enable proactive services and cross-platform workflows. For proactive service, the harness parses mobile notifications into structured events, associates them with persistent affairs, and prepares decision-ready proposals before the user asks. Refer to the figure below for a demonstration of a travel recovery plan triggered by a flight cancellation notification.

For cross-platform execution, the harness acts as a hierarchical planner-executor system, decomposing objectives into dependency-aware subtasks and coordinating parallel execution across mobile and computer environments while maintaining shared context.

Experiment

The evaluation setup spans mobile, computer, browser, DeepSearch, GUI grounding, and general agentic benchmarks, comparing three Qwen-UI-Agent variants against frontier proprietary models, open-weight foundation models, and specialized GUI agents. Each experiment validates a distinct capability: mobile-use tests demonstrate state-of-the-art long-horizon, cross-app execution on both simulated and real-device environments; computer-use evaluations show competitive performance through hybrid GUI and CLI action spaces with batched execution; browser and DeepSearch results confirm strong web navigation and evidence synthesis; GUI grounding benchmarks reveal accurate element localization across diverse interfaces; and general capability assessments indicate that GUI post-training preserves broad reasoning and agentic skills. Behavioral analyses further uncover that limited real-device experience leads to specific failure modes like exploration loops and pop-up interference, while action RL and online RL systematically correct step-level errors and foster trajectory-level patterns such as verified completion and cross-modal collaboration. Overall, Qwen-UI-Agent achieves leading or highly competitive results across all evaluated domains, with the 27B model offering a favorable balance of task capability, real-device reliability, and deployment efficiency.

Online RL shifts the agent from prematurely declaring success toward verifying outcomes before termination. This leads to more balanced use of Bash for execution and GUI for visual inspection, with emergent cross-modal collaboration that improves constraint satisfaction over long horizons. After online RL, trajectories containing at least one verification action increase by 14.7%, while the false-stop rate drops by 11.2%. The share of GUI actions rises by 6% and trajectories combining both GUI and Bash increase by 10.6%, with execution–verification transitions growing from 40.2% to 52.4%. Constraint satisfaction improves by 8.6% on OSWorld and 7.5% on BrowseComp-ZH after online RL.

On the GUI-only subset of MobileWorld, Qwen-UI-Agent-27B achieves the highest success rate, outperforming both general-purpose vision-language models and specialized GUI agents by substantial margins. Its smaller variant also surpasses all specialized baselines and most general-purpose models, demonstrating strong mobile task completion capability under a standard step budget. Qwen-UI-Agent-27B reaches a success rate of 82.1%, exceeding the next best general-purpose model by 8.9 percentage points. The strongest specialized GUI baseline, GUI-Owl-1.5-32B-Instruct, achieves 43.9%, which Qwen-UI-Agent-27B surpasses by 38.2 percentage points. Qwen-UI-Agent-35B-A3B attains 65.0%, outperforming all specialized GUI models and all general-purpose VLMs except Seed 2.1 Pro and Claude Opus 4.8. Among general-purpose VLMs, Seed 2.1 Pro leads with 73.2%, followed by GPT-5.6 Sol at 70.1% and Claude Opus 4.8 at 67.5%.

On real-device mobile benchmarks, the proposed model achieves strong performance, with the largest variant reaching the highest success rates on both MobileWorld-Real and AndroidDaily. It surpasses several closed-source general-purpose vision-language models and substantially outperforms all specialized GUI baselines, while the smaller variant also exceeds all specialized models and most general-purpose competitors. The largest model attains the top success rates on both MobileWorld-Real and AndroidDaily, outperforming closed-source models such as Seed 2.1 Pro, Gemini 3.1 Pro, GPT-5.6 Sol, and Claude Opus 4.8. The 27B model surpasses the strongest specialized GUI baseline, GUI-Owl-1.5-32B-Instruct, by a wide margin on MobileWorld. The smaller 35B-A3B variant achieves a success rate of 65.0% on MobileWorld, exceeding all specialized GUI baselines and all general-purpose vision-language models except Seed 2.1 Pro and Claude Opus 4.8. On AndroidDaily, several closed-source models reach success rates above 92%, with the proposed model leading at 95.2%. Open-source and smaller models, such as Qwen 3.7 Plus and Kimi K2.6, score considerably lower on both benchmarks, falling below 80% on MobileWorld-Real and below 80% on AndroidDaily.

Qwen-UI-Agent achieves a 79.5% success rate on OSWorld-Verified, ranking second overall and outperforming most leading closed-source and all evaluated open-weight foundation models. It trails only Claude Opus 4.8, while surpassing Seed 2.1 Pro, GPT-5.5, Gemini 3.5 Flash, Gemini 3.1 Pro, and Qwen 3.7 Plus. Qwen-UI-Agent attains the second-highest success rate among all compared models, behind only Claude Opus 4.8. It outperforms all evaluated open-weight models and several prominent closed-source alternatives, including GPT-5.5 and Gemini 3.5 Flash. The model's performance establishes a substantial lead over the strongest open-weight baselines on this benchmark.

On OSWorld-v2, Qwen-UI-Agent achieves a partial progress score of 40.0% and a binary completion rate of 13.9%, placing it competitively among frontier closed-source models. It substantially outperforms all open-weight models, improving partial progress by over 17 points and binary completion by over 9 points over the strongest open-weight baseline, while requiring far fewer steps per task. The batched-action design contributes to its efficiency, with an average of 135.8 steps compared to over 170 for single-action open-weight models. Qwen-UI-Agent exceeds GPT-5.5 in binary completion by 0.9 points, though it trails in partial progress by 9.5 points. It requires only 135.8 steps per task, far fewer than MiniMax M3 (326.7) and Qwen 3.7 Plus (173.5), demonstrating the efficiency of batched actions.

Online RL shifts the agent from prematurely declaring task completion toward verifying outcomes before termination, leading to more balanced use of Bash and GUI actions and emergent cross-modal collaboration that improves long-horizon constraint satisfaction. The Qwen-UI-Agent family achieves strong results across multiple benchmarks: on GUI-only MobileWorld, the 27B model substantially outperforms both general-purpose vision-language models and specialized GUI agents; on real-device mobile benchmarks, the largest variant attains top success rates and surpasses several closed-source models; on OSWorld-Verified, it ranks second overall behind only Claude Opus 4.8 while outperforming all evaluated open-weight models; and on OSWorld-v2, it achieves competitive scores among frontier closed-source models while substantially exceeding all open-weight baselines with greater efficiency due to its batched-action design.