One-sentence Summary

Researchers from Shanghai Jiaotong University and Zhejiang University propose SoftwareGym, an automated pipeline that synthesizes diverse, runnable GUI software environments with runtime-verified behavior via iterative GUI evaluation and repair, produces 182 applications across twelve categories with a 91.5% mean GUI task pass rate, constructs the 120-task SoftwareGym-Bench evaluation suite spanning the same categories, and yields a computer-use agent that improves Pass@3 from 41.39% to 56.97% on the strictly out-of-domain OSWorld benchmark and reaches 40.0% on SoftwareGym-Bench.

Key Contributions

• SoftwareGym is introduced as an automated design-implementation-evaluation pipeline that synthesizes structurally diverse standalone desktop applications and uses GUI evaluation with iterative repair to ensure functional reliability, achieving a mean GUI task pass rate of 91.5% across 182 applications in twelve categories.
• The released software library contains 182 runnable applications across twelve categories, and SoftwareGym-Bench provides a dedicated 120-task test set spanning the same twelve categories.
• Training only on trajectories from these synthesized environments, without using any OSWorld application data, improves Pass@3 on OSWorld from 41.39% to 56.97% and reaches 40.0% on SoftwareGym-Bench.

Introduction

Training computer-use agents requires interaction data from executable software environments that are diverse, realistic, and reliable. Prior approaches do not jointly satisfy these needs: native desktop and mobile applications preserve realistic states and complex workflows but are costly to integrate and risk application-specific overfitting, while browser-hosted environments scale well but their shared web substrate limits interaction depth and workflow complexity. The authors introduce SoftwareGym, an automated design-implementation-evaluation pipeline that synthesizes standalone desktop applications with controllable functional specifications, diverse interface and interaction designs, and combined source-level and GUI verification. The pipeline produces 182 runnable applications across twelve categories with a mean GUI task pass rate of 91.5%, and trajectories collected from these applications improve an agent's OSWorld Pass@3 from 41.39% to 56.97% without using any OSWorld applications, tasks, or trajectories for training.

Dataset

• Composition and sources: The dataset consists of synthetic GUI tasks built from retained software variants. Source materials are each variant’s functional manual, produced during implementation, and its GUI evaluation report, which identifies capabilities that have actually been verified in the running application.
• Capability filtering: The pipeline constructs an evidence-grounded function tree. A capability is marked supported only when it cites a passed case, checkpoint, or interaction step. Untested functions, failed functions, and blocked paths are excluded. Each tree leaf records the capability, support status, GUI entry points, constraints, observable state, and evaluation evidence.
• Difficulty subsets and task types: Tasks are assigned to easy, normal, and hard levels. Easy tasks have one primary goal with limited composition. Normal tasks combine related objects or operations with a necessary dependency. Hard tasks form multi-stage workflows in which each stage establishes state required by the next stage. Each difficulty level includes two task types: file-based tasks, which start with an existing input file, and blank-state tasks, which start from the application’s default GUI state. Unrelated or repeated actions cannot be added only to increase difficulty.
• Task generation and schema: Each planned workflow is converted into a natural-language user request and an internal evaluation specification. The user-facing request hides coordinates, detailed operation paths, capability identifiers, and evaluation mechanisms. The internal specification records the initial state, outcome criteria, process criteria, and expected final state. Deterministic validation rejects unsupported capabilities, inconsistent initial states, incompatible assets, and unreachable final states.
• Use in the paper: The resulting natural-language tasks have feasible, observable, and deterministically verifiable requirements. This section does not specify numeric subset sizes, training splits, or data mixture ratios; it describes how the task dataset is constructed and filtered.

Method

The authors present SoftwareGym, a pipeline for synthesizing diverse, reliable, and executable applications along with corresponding GUI tasks. The overall framework operates in three main stages: Design, Implementation, and Evaluation.

Refer to the framework diagram

In Stage A, the Software Design phase produces rich functional specifications and distinct interfaces. Given a software type and supporting resources, the FunctionDesigner expands the initial description through a four-step workflow. It establishes a functional baseline, extends modules with domain-specific professional functions, connects these functions into complete user workflows, and records each function as an implementation and evaluation contract in the Software Specification. Following this, the StyleDesigner generates multiple diverse UI blueprints that realize the same functionality with different interface designs. To ensure structural distinctness, the system computes pairwise tree-edit distances between normalized layout trees and regenerates blueprints that fall below a predefined threshold. Concurrently, the IconDesigner derives a style-independent Icon Candidate Manifest, and the SubIconDesigner renders them as style-specific SVG assets. These components are compiled into a Verified Design Package containing the Software Specification, UI Blueprint, and SVG Assets.

In Stage B, the Implementation phase turns each Verified Design Package into a runnable application. The Coder uses the package to realize the complete Avalonia application, including its structure, visual system, icons, and functionality. The DevCoder then audits the implementation against a functional manual, checking command bindings, application logic, and persistence, and repairs any missing or incomplete implementations.

In Stage C, the Evaluation phase tests the running application independently. The EvalTaskGenerateAgent generates evaluation cases from the source-verified functional manual. The GUI Agent executes these cases with mouse and keyboard actions, recording interaction traces and failure locations. The functional pass rate is calculated as $P_{\mathrm{func}} = N_{\mathrm{pass}} / N_{\mathrm{all}}$Pfunc​=Npass​/Nall​, where $N_{\mathrm{pass}}$Npass​ is the number of passed tasks and $N_{\mathrm{all}}$Nall​ is the total number of generated cases. If the application fails, the evaluation feedback is returned to the Implementation stage for in-place repair. Additionally, a normalized visual-fidelity score ensures that implementation repairs preserve the intended interface structure. Applications that pass all cases and maintain design fidelity enter the final software library.

Once the software variants are verified, the system proceeds to Task Synthesis to convert their verified functions into executable GUI tasks.

As shown in the figure below:

Task synthesis consists of three steps: function analysis, difficulty leveling, and task generation. First, Function Analysis identifies application capabilities verified through GUI interaction. It extracts the functional manual and uses the GUI evaluation report to construct an evidence-grounded function tree. A capability is marked as supported only when it cites a passed case or interaction step, preventing task generation from relying on untested functions.

Next, Difficulty Leveling controls task complexity through dependencies among verified capabilities. Tasks are assigned to three difficulty levels: easy, normal, and hard. An easy task involves a single primary goal with limited functional composition. A normal task combines related objects or operations with necessary dependencies. A hard task forms a multi-stage workflow where each stage establishes the state required by the next. Each difficulty level includes both file-based tasks, which start with an existing input file, and blank-state tasks, which operate from the default GUI state.

Finally, Task Generation converts each planned workflow into a natural user request and an executable evaluation specification. It validates the selected capabilities against the manual, passed GUI evidence, and state dependencies. The resulting user-facing request describes a single user goal without exposing coordinates or detailed operation paths, while the internal task specification records the initial state, outcome criteria, and expected final state. Deterministic validation ensures that the produced natural-language tasks remain feasible, observable, and deterministically verifiable.

Experiment

The experiments evaluate SoftwareGym, a pipeline that synthesizes applications with controlled visual style, implements them, and iteratively repairs them using GUI-guided testing. A 9B model trained only on SoftwareGym trajectories improves OSWorld Pass@3 from 41.39% to 56.97% and SoftwareGym-Bench Pass@3 from 25.8% to 40.0%, showing transferable and in-domain gains. The library provides 182 runnable applications across twelve categories, with evaluation-guided repair raising average GUI pass rates from 54.9% to 91.5%, while ablations confirm that style design increases structural diversity and GUI feedback is essential for reliable repair.

On OSWorld, API foundation agents show application-dependent performance, with several models scoring much higher on Calc and VLC than on Chrome, and Gemini 3.1 Pro and Seed models leading overall while GPT-5.5 trails. The cited analysis reports that SoftwareGym-only supervision improves every evaluated OSWorld domain and substantially raises overall Pass@3 without any OSWorld training data. Among API foundation agents, Gemini 3.1 Pro and Seed 2.1 Pro achieve the leading overall OSWorld scores, while GPT-5.5 records the lowest overall in this group. SoftwareGym-only supervision transfers to all evaluated OSWorld domains and raises overall Pass@3 from 41.39% to 56.97% without using OSWorld training data.

On SoftwareGym-Bench, API foundation agents show uneven domain performance, with GPT-5.4 and GPT-5.5 leading overall and Seed models trailing. Tasks involving mail and browser interactions are generally stronger, while music and slides remain weaker. The associated analysis notes that SoftwareGym supervision lifts a compact 9B model to 40.0% overall, ahead of Qwen3.7-Plus and several larger specialized CUAs. GPT-5.4 and GPT-5.5 achieve the highest overall pass rates among API foundation agents, while Seed 2.0 Pro and Seed 2.1 Pro rank lowest overall. Performance is typically strongest in mail and browser categories and weakest in music and slides across the listed models. SoftwareGym supervision improves the same 9B base model by over 14 points and produces competitive in-domain results at a compact scale.

Different training-software sources affect OSWorld and SoftwareGym-Bench performance unevenly. Training only on SoftwareGym applications is competitive on several individual OSWorld apps but lower on overall OSWorld than training only on OSWorld data. Adding OSWorld trajectories to SoftwareGym data improves SoftwareGym-Bench results while overall OSWorld performance remains slightly below the OSWorld-only setting. SoftwareGym-only training outperforms OSWorld-only training on Chrome and VS Code but underperforms on Calc, Impress, Writer, and VLC, leading to lower overall OSWorld pass@3. Combining OSWorld and SoftwareGym trajectories improves SoftwareGym-Bench from 40.0% to 47.5%, while overall OSWorld drops from 61.48% to 58.61% compared with OSWorld-only training.

These experiments evaluate API foundation agents on OSWorld and SoftwareGym-Bench and examine how training-software sources transfer. On OSWorld, performance varies by application, with Gemini 3.1 Pro and Seed models generally leading while GPT-5.5 trails, and SoftwareGym supervision improves all OSWorld domains without using OSWorld training data. On SoftwareGym-Bench, GPT-5.4 and GPT-5.5 lead while Seed models lag, and SoftwareGym supervision helps a compact 9B model surpass larger agents, especially in mail and browser tasks. Comparisons of training sources show that SoftwareGym-only training transfers unevenly across OSWorld apps, and adding OSWorld data improves SoftwareGym-Bench but slightly lowers overall OSWorld relative to OSWorld-only training.