One-sentence Summary

Tencent and Fudan University introduce WORKFORGE, a scalable synthesis framework that avoids costly handcrafted or oversimplified synthetic setups by converting expert workflows, retrieved real-world files, and workspace-derived factual anchors into $16.7\text{K}$16.7K verifiable work-agent environments spanning $40$40 professional domains and $60$60 file types, enabling Qwen3.5-35B-A3B-Base post-training to improve GDPval from $45.5$45.5 to $73.6$73.6 and APEX Score from $5.0$5.0 to $21.3$21.3 while helping Qwen3.5-27B achieve highly competitive results.

Key Contributions

• WORKFORGE is introduced as a scalable framework that constructs verifiable work-agent environments from real-world resources by profiling expert workflows, retrieving relevant files, and organizing audited workspaces whose observable facts anchor task specifications, solution plans, and verification criteria. It derives each task’s instructions, plans, and complementary programmatic and semantic verifiers from these factual anchors, keeping verification tied to workspace evidence.

• The framework instantiates 16.7K verifiable environments across 40 professional domains, with workspaces collectively covering 60 file types and rich cross-file dependencies. These environments support long-horizon interaction and produce supervision for agent post-training.

• Fine-tuning Qwen3.5-35B-A3B-Base on the resulting trajectories improves GDPval from 45.5 to 73.6 and APEX from 5.0 to 21.3 across five benchmarks, while Qwen3.5-27B reaches competitive performance. Controlled analyses show consistent gains as data volume and interaction horizons increase.

Introduction

Autonomous agents are moving large language models from conversational assistants toward digital workers that complete professional workflows across files and tools. This shift matters because real deliverables depend on authentic workspaces with complex file dependencies, so training environments must expose realistic friction and verifiable feedback. Prior environment synthesis often relies on synthetic or mock resources, creating toy workspaces with limited diversity and cross-file relationships, while ungrounded evaluation can introduce hallucinated or unsupported requirements. The authors introduce WORKFORGE, a scalable framework that builds verifiable work-agent environments from real-world digital resources by grounding task specifications and verifier rubrics in observable workspace facts. The authors leverage expert workflows as blueprints, retrieve and profile real resources to recover file roles, dependencies, and factual anchors, and then generate tasks plus programmatic and semantic checks for reliable trajectory collection. The framework yields 16.7K environments across 40 domains, and fine-tuning Qwen3.5-35B-A3B-Base on these trajectories improves performance on several agent benchmarks.

Dataset

• The authors describe a synthesized agentic dataset built from scene blueprints that are expanded into concrete software workspaces.

• Each scene blueprint is mapped to multiple environments by acquiring distinct resource pools from external sources. An acquisition agent searches, reads, downloads, and previews resources to keep only relevant artifacts and associate them with the abstract requirements of the blueprint.

• Each retained environment contains a workspace realized from one nonempty resource pool. The workspace preserves the original file organization of the selected pool, and the number of retained environments per blueprint is denoted by K_j.

• Each workspace is paired with an environment profile. The profile summarizes the domain, file roles, and possible task directions; records observable workspace facts; and stores artifact schemas and statistics. These profiles provide factual anchors for task construction and verification.

• The dataset instance used for agent evaluation includes the visible workspace, task instructions, trajectory records, output artifacts, programmatic scores, semantic scores, and model configuration. Trajectories are produced by letting a solver agent interact with the workspace until it submits deliverables or reaches an execution limit.

• After rollout, programmatic and semantic graders evaluate the final workspace and submitted outputs. Complete execution records are retained as scored trajectory data, linking agent behavior to verifiable outcomes.

• The paper refers to corpus-level scalar statistics, domain coverage, file types, task scores, and tool-call distributions in its reporting, but the provided excerpt does not specify exact subset sizes, training splits, mixture ratios, cropping rules, or metadata schema values.

Method

The authors introduce WORKFORGE, a framework designed to translate professional workflows into realistic and verifiable agent environments. This approach ensures that agent environments scale without sacrificing authenticity. The overall framework operates through a four-stage pipeline, as illustrated in the figure below.

Workflow Specification To generate sophisticated professional tasks, the authors engage domain experts to distill their workflows into reusable Scene Blueprints. These blueprints identify dependencies between initial conditions, domain-specific operations, and final deliverables. A collection of $M$M blueprints is defined as $\mathcal{B} = \{b_j\}_{j=1}^M$B={bj​}j=1M​, where each blueprint $b_j = (\mathcal{W}_j, \mathcal{O}_j, \mathcal{R}_j, \mathcal{D}_j)$bj​=(Wj​,Oj​,Rj​,Dj​) specifies the workflow context $\mathcal{W}_j$Wj​, the expected deliverable $\mathcal{O}_j$Oj​, required sources and dependencies $\mathcal{R}_j$Rj​, and domain-specific decisions $\mathcal{D}_j$Dj​. Each blueprint serves as a seed connecting expert intent to task generation.

Environment Synthesis This stage integrates environment mapping and environment profiling to instantiate each blueprint into a concrete workspace. For a scene blueprint $b_j$bj​, the stage produces a set of environments $\mathrm{Env}(b_j) = \{(W_{jk}, P_{jk})\}_{k=1}^{K_j}$Env(bj​)={(Wjk​,Pjk​)}k=1Kj​​.

Environment mapping expresses the one-to-many relationship between a blueprint and its concrete environments. An acquisition agent iteratively searches for sources and downloads resources to form distinct resource pools $\{R_{jk}\}_{k=1}^{K_j}${Rjk​}k=1Kj​​. The workspace is realized as $W_{jk} = \phi_{R_{jk}}(b_j)$Wjk​=ϕRjk​​(bj​).

Environment profiling defines the description accompanying each environment. The profile $P_{jk} = (U_{jk}, \mathcal{F}_{jk}, A_{jk})$Pjk​=(Ujk​,Fjk​,Ajk​) includes a summary $U_{jk}$Ujk​ of the domain and file roles, workspace facts $\mathcal{F}_{jk}$Fjk​, and artifact schemas and statistics $A_{jk}$Ajk​. This profile supplies factual anchors for task construction and verification.

Task Generation Task generation produces executable task packages with aligned requirements and verification criteria.

Task Design combines the scene blueprint and environment profile to specify the professional objective. The task designer jointly generates the visible instruction $q_{jk}$qjk​ and the hidden solution plan $h_{jk}$hjk​:

Verifier Design translates these requirements into an atomic rubric set $\mathcal{V}_{jk} = \mathcal{V}_{jk}^{\text{prog}} \cup \mathcal{V}_{jk}^{\text{sem}}$Vjk​=Vjkprog​∪Vjksem​. Programmatic criteria check final artifacts $Y_{jk}$Yjk​ to produce a score $S_{jk}^{\text{prog}}$Sjkprog​. Semantic criteria configure an LLM judge $G_{jk}^{\text{sem}}$Gjksem​ to assess open-ended requirements, producing a normalized score $S_{jk}^{\text{sem}}$Sjksem​. The resulting task package separates visible inputs from hidden components:

Trajectory Generation Trajectory generation turns the static task package into observed agent behavior. For each task package $\mathcal{T}_{jk}$Tjk​, a solver agent $\pi_\theta$πθ​ receives only the visible pair $(W_{jk}, q_{jk})$(Wjk​,qjk​) and interacts with the workspace. This rollout process is denoted as:

where $\tau_{jk}$τjk​ records ordered messages and tool interactions, and $Y_{jk}$Yjk​ denotes output artifacts. After execution, the graders evaluate the final workspace and artifacts to produce programmatic and semantic scores. The complete execution is retained as a scored trajectory record $z_{jk} = (\tau_{jk}, Y_{jk}, S_{jk}^{\text{prog}}, S_{jk}^{\text{sem}}, \mu_{jk})$zjk​=(τjk​,Yjk​,Sjkprog​,Sjksem​,μjk​). The resulting corpus exhibits diverse professional domains, file types, and tool usage patterns, as summarized in the figure below.

Experiment

The experiments evaluate WORKFORGE-trained agents on several work-agent benchmarks against proprietary, large open-source, and smaller open-source baselines, while ablations examine the effects of training harness, verification signals, and workspace authenticity. Overall, the results show that the proposed training pipeline produces strong generalist work agents, with the smaller model performing well among sub-100B baselines and the larger model approaching frontier capability, especially on tasks requiring complex workspace exploration and evidence integration. Further analyses indicate that broader real-world data, longer interaction trajectories, aligned evaluation harnesses, and hybrid verification each contribute to more sustained tool use, better planning, and higher-quality professional deliverables.

WORKFORGE provides a large collection of tasks with multiple evaluation rubrics per task and heterogeneous workspaces containing many files and file types. It also records trajectories that include tool interactions, output artifacts, and programmatic and semantic scores. These task, workspace, and trajectory dimensions support analysis of agent behavior and scaling studies. The dataset includes a large number of tasks with multiple rubrics per task. Workspaces are file rich and diverse, averaging many files and many file types. Trajectory records capture ordered agent interactions, output artifacts, and evaluation scores. Real world acquisition enlarges workspace scale and heterogeneity, providing a richer artifact and evidence space for agents.

WORKFORGE-A3B is competitive across agent capability benchmarks, with especially strong gains over prior specialized sub-100B models on GDPval and continued strength under strict pass-rate metrics. WORKFORGE-27B also improves a larger open model across WildClaw, ClawEval, GDPval, APEX, and ALE, reaching frontier-like GDPval performance with far fewer parameters. WORKFORGE-A3B surpasses prior specialized sub-100B baselines by a large margin on GDPval while staying competitive on WildClaw, ClawEval, APEX, and ALE. WORKFORGE-27B improves across all five benchmarks and narrows the gap to frontier systems such as Claude and GPT on GDPval.

Hybrid trajectory selection combines LLM-based quality verification with programmatic checks and performs better than using either signal alone. This suggests that verification signals from both exact workspace states and content quality provide more useful supervision for selecting training trajectories. Hybrid verification produces the strongest results across the reported metrics. LLM-only and programmatic-only selection remain similar and fall below the combined method.

The experiments evaluate WORKFORGE’s dataset design, model capability on agent benchmarks, and trajectory selection strategies. The dataset’s many tasks, rubrics, diverse workspaces, and detailed trajectory records support analysis of agent behavior and scaling trends. WORKFORGE-trained models show competitive or improved performance across agent evaluation suites, with especially strong gains on GDPval and reduced gaps to frontier systems. Hybrid trajectory selection using both programmatic and semantic verification outperforms either signal alone, indicating that combining workspace-state checks with content-quality evaluation provides better supervision.