One-sentence Summary

Researchers from MMLab, CUHK, et al. propose CUA-Master, a nested double-loop recursive improvement framework that pairs an outer world-data-model flywheel with an inner open-world seeding and autocatalytic propagation loop and adds a feedback-guided snapshot-rollback retry mechanism, enabling CUA-Master-27B to score 80.2 on OSWorld Verified and improve data synthesis across recursive iterations.

Key Contributions

• CUA-Master is introduced as a nested double-loop recursive improvement framework whose outer world-data-model flywheel incorporates open-world corpus and uses evolving model weaknesses to guide iterative data construction and optimization.
• Its inner data flywheel couples open-world seeding from instructional videos with autocatalytic propagation, reconstructing trajectories via a visual-semantic dual-path mechanism and abstracting seeds into reusable primitives for controllable synthesis of diverse, complex, and verifiable tasks; a feedback-guided snapshot-rollback retry mechanism supports reliable and efficient trajectory collection.
• Experiments show that CUA-Master-27B achieves scores of 80.2 on OSWorld Verified, 36.9 on OSWorld 2.0, 43.9 on WeaveBench, and 25.0 on Agents’ Last Exam, while the framework improves data synthesis quality and efficiency and yields continuous gains across recursive iterations.

Introduction

Computer-use agents (CUAs) are emerging as a way to automate real-world workflows across graphical interfaces, command-line tools, and long-horizon cross-application tasks. Prior work is limited by unidirectional data generation and model training pipelines, while recursive self-improvement methods often remain closed-world, relying mostly on the model’s own knowledge and reinforcing existing biases. Synthetic data further suffers from limited diversity, weak controllability over task complexity and dependencies, and low rollout reliability at scale. The authors introduce CUA-Master, a nested double-loop recursive framework that combines an outer world-data-model flywheel with an inner data flywheel to support open-world exploration, visual-semantic trajectory reconstruction, primitive-based task composition, and feedback-guided snapshot-rollback retries. This design aims to continuously incorporate external knowledge and use evolving model weaknesses to guide data generation, with a 27B model achieving competitive results on OSWorld, WeaveBench, and Agents’ Last Exam.

Dataset

The authors construct the dataset as a generated and filtered corpus rather than a fixed collection: open-world instructional videos are harvested, converted into seed tasks and reconstructed trajectories, then expanded into synthetic tasks and verified rollouts.

Sources and composition

• Videos are retrieved from YouTube, TikTok, and Bilibili.
• The search space is organized through a hierarchical application taxonomy with 8 main categories, 36 subdomains, and over 650 applications.
• Queries combine application names with operation-oriented keywords such as “tutorial”, “how to”, and “guide”.
• Retrieved videos are deduplicated by source identity and normalized title.
• Metadata gates enforce duration, upload recency, source availability, and taxonomy-consistent application attribution.
• A lightweight noise filter suppresses promotional, entertainment, language-learning, and incidental software content that does not show computer-use interactions.
• Uniformly sampled frames are transcribed with Whisper-Large-v3 through Faster-Whisper, producing timestamped transcripts and taxonomy metadata.
• Annotator A, normally a more powerful model, assigns a CUA suitability score from 0 to 5 and predicts CUA presence, trajectory availability, and visual clarity.

Seed task mining

• Videos are segmented into multimodal temporal episodes by Annotator A.
• Each episode has start and end timestamps and a task description.
• The pipeline keeps only observable computer-operation state-continuous intervals.
• Introductions, advertisements, talking-head commentary, slide-only explanations, and other non-operational content are excluded.
• A single tutorial video can yield multiple independent seed tasks.
• Each seed task receives a concise goal inferred from visual context, narration, application identity, and demonstrated state transition.

Trajectory reconstruction and processing

• No explicit spatial cropping strategy is described; visual processing uses uniformly sampled frames and frame-level appearance-change scoring.
• A semantic path detects operation-bearing clauses such as “click”, “press”, “scroll”, and “drag”.
• A visual path uses FFmpeg to detect frame-level appearance changes indicating interface transitions such as menu openings, dialog appearances, page navigation, or content updates.
• Overlapping semantic and visual triggers within a temporal window become trajectory boundary candidates.
• Annotator A acts as an action rechecker, action annotator, and trajectory verifier.
• Final actions include interaction coordinates, typed content, and other keys.
• Duplicated actions, non-operational changes, and substantial temporal or semantic discontinuities are rejected.

Primitive library and synthetic task generation

• Seed tasks and trajectories are converted into reusable application-level ability primitives.
• Each primitive contains the associated application, reusable ability, execution preconditions, environment requirements, and verifier specification.
• Extracted primitives are aggregated into an application-indexed primitive library.
• Semantically equivalent primitives are canonicalized and deduplicated, while variants with different preconditions, environment requirements, or verification logic are retained.
• Synthetic tasks are created by composing compatible primitives under a complexity budget and a chosen number of applications.
• Single-application tasks are generated when one application is used; cross-application tasks are generated when more than one application is used.
• High-quality synthetic tasks are added to the task set used for a particular model training stage.

Trajectory collection and usage

• For each task, an environment is materialized from a VM image pool and a trajectory is rolled out.
• If verification fails or inefficiency is detected, Annotator A estimates the earliest state-divergence step and generates execution-grounded feedback.
• Feedback describes failures or inefficiencies without exposing privileged action solutions.
• The environment is restored to the latest valid snapshot, and the policy keeps the valid prefix, discards the undesirable suffix, and resumes with feedback as hints.
• Only successful and most efficient trajectories are retained as final supervision.
• The provided excerpt does not specify the final dataset size or exact training/validation mixture ratios.

Method

Method

The authors formalize interactive computer-use tasks as a tuple

$$\mathcal{T} = (\mathcal{I}, \mathcal{E}, \mathcal{V}),$$T=(I,E,V),

where $\mathcal{I}$I is the natural-language instruction, $\mathcal{E}$E is the interactive environment, and $\mathcal{V}$V is a task-specific verifier that judges whether the objective has been achieved. At each execution step $s$s, the agent receives an observation $o_s$os​ and produces an action sequence

$$\mathbf{a}_s \sim \Pi(\cdot \mid \mathcal{I}, o_s, h_s),$$as​∼Π(⋅∣I,os​,hs​),

where $\Pi$Π is the agent policy, $h_s$hs​ is the interaction history, and $\mathbf{a}_s = (a_s^{(1)}, \dots, a_s^{(L_s)})$as​=(as(1)​,…,as(Ls​)​) is a batch of $L_s$Ls​ atomic actions. The resulting trajectory is

$$\mathcal{X} = \{o_1, \mathbf{a}_1, \dots, o_S, \mathbf{a}_S, o_{S+1}\},$$X={o1​,a1​,…,oS​,aS​,oS+1​},

which is later evaluated by $\mathcal{V}$V.

Hybrid Action Space and Environment Infrastructure

The action space unifies GUI operations and structured CLI operations. GUI actions consist of atomic mouse and keyboard operations, while CLI actions execute shell commands. The two modalities can be interleaved within a single trajectory, allowing the policy to choose the most suitable interface for the current state. GUI observations are grounded through screenshots, and CLI observations are grounded through standardized shell outputs containing command output, error streams, and execution status. Pointer coordinates are represented in a resolution-independent normalized coordinate system, and the action interface also supports tool invocation.

The rollout infrastructure is a backend-agnostic, profile-driven orchestration layer for reproducible tasks across Linux, Windows, and macOS. Virtual machine images are assembled through a semi-automated pipeline: base images are prepared manually, then additional variants are generated programmatically by removing applications or reorganizing desktop layouts. Each image is registered in a catalog with operating system, version, and application metadata. A resource-aware scheduler bounds concurrency by VM slots, and lifecycle hooks manage resource acquisition, controller readiness, network tunneling, reset, and cleanup. Episodes run on immutable base images with disposable writable overlays, enabling harness bootstrapping, task initialization, and transient application state.

Recursive Flywheel

The overall data generation and training process is organized as a recursive flywheel with two complementary forces. Open-world seeding injects real-world diversity by discovering underrepresented applications, operations, and workflows. Autocatalytic propagation transforms these seeds into reusable primitives and composes them into executable tasks and trajectories. A broader world-data-model flywheel connects open-world exploration, data construction, model improvement, and feedback-driven re-exploration, so that model failures guide subsequent data acquisition toward underrepresented workflows and harder tasks.

Open-World Seeding

Open-world seeding distills instructional videos into seed tasks and reconstructed trajectories. The process begins with corpus harvesting. A hierarchical application taxonomy and operation-oriented keywords are used to query internet video sources. Retrieved videos are deduplicated, filtered by metadata such as duration and source availability, and scored by an annotator model for computer-use suitability, interaction clarity, and trajectory reconstructibility.

Seed task mining then segments a video into independently executable tasks. Given timestamped transcripts and sampled frames, the annotator model decomposes the video into multimodal temporal episodes:

$$\{\hat{f}_1, \dots, \hat{f}_n\} \xrightarrow{\mathcal{A}} \{(t_i^s, t_i^e, \mathcal{T}_i)\}_{i=1}^{M},$${f^​1​,…,f^​n​}A​{(tis​,tie​,Ti​)}i=1M​,

where $t_i^s$tis​ and $t_i^e$tie​ denote episode boundaries and $\mathcal{T}_i$Ti​ is a concise inferred goal. Non-operational fragments such as introductions, advertisements, talking-head commentary, and slide-only explanations are discarded.

For each seed task, a dual-path trajectory reconstruction mechanism recovers the action sequence. The semantic path splits transcripts into atomic clauses and identifies operation-bearing expressions such as click, press, scroll, and drag. The visual path detects frame-level appearance changes that indicate interface transitions. The two signals are aligned into a boundary lattice:

$$\mathcal{C} = \left\{(t_i^{\mathrm{sem}}, t_j^{\mathrm{vis}}) \mid |t_i^{\mathrm{sem}} - t_j^{\mathrm{vis}}| \leq T \wedge t_i^{\mathrm{sem}} \in \mathcal{C}^{\mathrm{sem}} \wedge t_j^{\mathrm{vis}} \in \mathcal{C}^{\mathrm{vis}}\right\}.$$C={(tisem​,tjvis​)∣∣tisem​−tjvis​∣≤T∧tisem​∈Csem∧tjvis​∈Cvis}.

The annotator model then rechecks candidates using local transcript context and paired pre-action and post-action frames, rejects non-operational changes, refines action boundaries, annotates accepted candidates as CUA actions, and verifies the final trajectories.

Autocatalytic Propagation

Autocatalytic propagation addresses the limited coverage and potential annotation noise of seed data. Its first stage is primitivization. The annotator model extracts reusable ability primitives from seed tasks and trajectories:

$$\left(\mathcal{T}^{\mathrm{seed}}, \mathcal{X}^{\mathrm{seed}}\right) \xrightarrow{\mathcal{A}} \{p_1, \dots, p_n\}, \quad p_i = (u_i, a_i, \mathcal{P}_i, \mathcal{E}_i, \mathcal{V}_i),$$(Tseed,Xseed)A​{p1​,…,pn​},pi​=(ui​,ai​,Pi​,Ei​,Vi​),

where $u_i$ui​ is the associated application, $a_i$ai​ is the reusable ability, $\mathcal{P}_i$Pi​ is a precondition, $\mathcal{E}_i$Ei​ is an environment requirement, and $\mathcal{V}_i$Vi​ is a verifier specification. Extracted primitives are canonicalized, deduplicated, and stored in an application-indexed primitive library.

Task synthesis is formulated as constrained primitive composition. Given a complexity budget $K$K and an application set $\{u_1, \dots, u_N\}${u1​,…,uN​}, the method controls task difficulty through the number of composed primitives and the number of involved applications. Compatibility between preconditions and environment requirements is checked before composition. The annotator model instantiates a coherent instruction and consolidates the component environment and verifier specifications:

$$\left(\mathcal{B}(\{u_1, \dots, u_N\}), K\right) \xrightarrow{\mathcal{A}} \mathcal{T}^{\mathrm{syn}} = \left(\mathcal{I}^{\mathrm{syn}}, \mathcal{E}^{\mathrm{syn}}, \mathcal{V}^{\mathrm{syn}}\right).$$(B({u1​,…,uN​}),K)A​Tsyn=(Isyn,Esyn,Vsyn).

This supports both single-application tasks and cross-application tasks.

The second stage is resilient trajectory collection. Instead of restarting failed rollouts from scratch, the method uses feedback-guided snapshot-rollback retries. When verification fails or inefficiency is detected, the annotator model inspects the verifier residuals, collected trajectory, and executable environment:

$$(\mathcal{T}, \mathcal{X}, \mathcal{E}) \xrightarrow{\mathcal{A}} (\hat{s}, \mathcal{F}),$$(T,X,E)A​(s^,F),

where $\hat{s}$s^ estimates the earliest state-divergence step, and $\mathcal{F}$F is execution-grounded diagnostic feedback without exposing privileged actions. The environment is restored to the latest valid snapshot before $\hat{s}$s^, the stable prefix is preserved, and the agent resumes execution conditioned on the feedback. Retries continue until verification passes or a retry budget is exhausted. Only the successful and most efficient branch is retained, so failed and inefficient suffixes are excluded from supervision.

Model Training

Training is integrated into the recursive flywheel. Early iterations use supervised fine-tuning on verified trajectories collected through both open-world seeding and autocatalytic propagation. At iteration $i$i, the training set aggregates data from all preceding iterations. The policy is optimized with an autoregressive objective over executable action sequences conditioned on instructions and interaction histories. To reduce optimization drift, each SFT iteration starts from the same base checkpoint with the accumulated updated data.

As SFT gains saturate, the final training stage switches to reinforcement learning. The SFT-trained policy is further optimized in real executable computer environments on high-quality synthetic tasks. Because task-specific verifiers are already constructed during task synthesis, outcome-based rewards are derived directly from execution results without manual reward annotation. This stage targets long-horizon decision-making, execution robustness, and task completion.

Experiment

Across four challenging CUA benchmarks spanning GUI-only and hybrid GUI+CLI settings, along with additional Windows and macOS benchmarks, CUA-Master-27B is compared against generalist and specialist baselines under a consistent evaluation protocol. The results show that it matches or surpasses larger and proprietary models on GUI-only tasks, delivers particularly strong gains on hybrid tasks, and achieves comparable performance with substantially lower runtime. Further analyses confirm that the hybrid GUI and CLI action space improves both task scores and execution efficiency, while the data flywheel yields broad task coverage and higher rollout success, and iterative world-data-model training produces progressive gains, especially on long-horizon benchmarks after reinforcement learning.

CUA-Master-27B outperforms the similarly sized UI-Mate-27B specialist across all four evaluated benchmarks, with the largest gains on WeaveBench and Agents' Last Exam. Among the Qwen generalist baselines, Qwen-3.8-Max achieves the highest OSWorld scores, while efficiency varies across models. CUA-Master-27B also shows strong hybrid CUA performance, trailing only a few leading models on the most challenging benchmark and staying competitive with much larger models on GUI-only tasks. CUA-Master-27B improves over UI-Mate-27B on all evaluated benchmarks, especially on WeaveBench and Agents' Last Exam. Qwen-3.8-Max leads the listed generalist models on OSWorld Verified and OSWorld 2.0, while Qwen-3.8-27B improves substantially on OSWorld 2.0 but remains lower on WeaveBench. CUA-Master-27B outperforms models derived from Qwen-3.5-27B in hybrid CUA scenarios and is competitive with the much larger Qwen-CUA-397B-A17B on GUI-only benchmarks.

CUA-Master-27B demonstrates strong cross-platform generalization, posting the best MacOSWorld result among the compared methods and the second-best MacAgentBench result. Its WindowsAgentArena performance is slightly below the top model but remains close to the leading group. Overall, it maintains competitive scores across both Windows and macOS benchmarks. CUA-Master-27B achieves the highest MacOSWorld score, narrowly outperforming the next best model. On MacAgentBench, it ranks second overall while staying ahead of the other 27B to 32B scale models listed. Its WindowsAgentArena result is competitive with the top entries, trailing only the best-performing model.

Rollout success rates for the proposed method exceed the baseline across every reported difficulty setting, from simple single-application tasks to complex multi-application tasks. The gap widens as task complexity increases, with the baseline dropping sharply while the proposed method retains substantially higher success rates. In the setting comparable to Agents' Last Exam, the proposed method reaches nearly 50% success versus 14.6% for the baseline. Success rates decline for both methods as the number of primitives and applications increases, but the baseline degrades much more steeply. At moderate complexity with 40 primitives across 3 applications, the proposed method achieves roughly 50% success, more than three times the baseline. Under the most challenging listed conditions, the proposed method maintains a success rate above 30% while the baseline falls below 2%.

The evaluation compares CUA-Master-27B with similarly sized specialist models, Qwen generalist baselines, and much larger CUA systems across GUI-only, hybrid CUA, and cross-platform Windows and macOS benchmarks. CUA-Master-27B consistently outperforms UI-Mate-27B, leads or stays competitive among comparable models, and remains effective relative to larger models. Rollout experiments further show that its advantage over the baseline increases with task difficulty, particularly on complex multi-application and high-primitive settings where the baseline degrades sharply.