One-sentence Summary

A team from Huawei, Shanghai Jiao Tong University, Northwestern University, et al. proposes a two-level framework that factorizes agentic data as $(E, q, \tau, v)$(E,q,τ,v) and formulates generation through the Accuracy–Complexity–divErsity (ACE) lens, shifting the focus from data quantity to the continual allocation of valid, informative, and non-redundant experience for evolving LLM agents.

Key Contributions

• A factorized representation of agentic data as a tuple (environment, task, interaction, verifier) unifies generation paradigms by their primary anchor and dependency structure, covering forward and reverse pipelines.
• The Accuracy–Complexity–Diversity (ACE) lens formulates generation as constrained distribution design: Accuracy defines the feasible support, Complexity calibrates difficulty relative to the learner, and Diversity controls behavioral coverage beyond surface variation.
• Applying this framework to the literature reveals a shift toward execution-grounded accuracy, learner-relative complexity, and behavioral diversity, and reframes agentic data generation as a continual process of allocating valid, informative, and non-redundant experience as agents and environments evolve.

Introduction

Large language model agents are increasingly deployed to act in environments by invoking tools, modifying files, or interacting with simulated worlds, which requires multi-turn experience connecting decisions to observations. Generating such interaction data at scale is essential for training and evaluation, but the core challenge is not just producing more data; it is ensuring that generated experience is accurate, appropriately complex for the learner, and sufficiently diverse. Prior work remains fragmented because pipelines are described in domain-specific terms (API calls, GUI demonstrations, simulator rollouts) and often mix how candidate data are constructed with how they are verified and selected, making it difficult to compare mechanisms across applications. The authors address this by introducing a unified factorization of agentic data into environment, task, trajectory, and verification components, and they propose the ACE framework (Accuracy, Complexity, Diversity) as a common objective for shaping generated data distributions. They also contribute a mechanism-oriented taxonomy that organizes generation paradigms by their primary anchor and dependency structure rather than by domain labels.

Dataset

The authors construct a diverse agent-training dataset by combining multiple source types and applying coverage-guided curation. The dataset is not a single static collection; it grows and rebalances as the model improves, so scale is measured by the marginal addition of valid capabilities rather than raw example count.

Dataset composition and sources

• Real-world traces: heterogeneous documents, interaction logs, and authentic long-tail support from actual user sessions or tool usage.
• Model-generated data: synthetic tasks and environments produced by teacher models, offering controllability but risking teacher priors.
• Programmatic environments: simulation-ready worlds, API ecosystems, executable repositories, and embodied scenes built through code, enabling reset and execution but potentially repeating a small set of templates.

Key details for each subset

• Real traces: sourced from documents, repositories, and user interactions; inherit natural source imbalance; filtered through verification regimes to keep only robustly valid examples.
• Synthetic generation: created by prompting models; controllable in volume and style; filtered to avoid repeating the teacher’s biases and to ensure task validity.
• Programmatic environments: constructed via tool-oriented pipelines and software-agent frameworks; verified through execution; a smaller set of thoroughly tested environments is preferred over a larger unreliable one.

How the paper uses the data

• The dataset is used for training agents, with no fixed split. Instead, the authors employ a model-adaptive pipeline: new experience is generated near the learner’s current frontier based on failures, uncertainty, or a target solve-rate band.
• Coverage-guided balancing removes redundancy, targets under-represented factor combinations, and allocates generation toward named transfer gaps.
• Stratified replay and periodic broad exploration preserve easier anchor cases and prevent overfitting to transient model weaknesses.

Processing and curation details

• Source expansion adds environments, tools, modalities, and verification regimes to increase diversity.
• Coverage-guided pipelines filter and rebalance the pool: they remove redundant examples, ensure all factor combinations are represented, and generate tasks specifically where the model struggles.
• Self-evolving loops feed observed behavior and world-model errors back into task generation, while trajectory-aware allocation explicitly targets under-covered reasoning and interaction structures.
• No cropping strategy is mentioned; metadata construction focuses on factor combinations and verification status to guide balancing.

Method

The authors model tool-augmented interaction as a partially observable Markov decision process. An agent chooses actions based on observable history, and the environment returns new observations while potentially changing its latent state. To standardize data generation across domains, the framework defines a common data object $d = (E, q, \tau, v)$d=(E,q,τ,v). Here, $E$E represents the environment specification containing tools, state, and rules; $q$q is the task signal; $\tau$τ is the interaction realization or trajectory; and $v$v is an optional verifier or reward interface.

This factorization clarifies that agentic data generation is not merely instruction generation followed by response sampling. Instead, the pipeline must establish compatible relations among an actionable environment, a grounded task, an interaction process, and a trustworthy success signal.

Agentic data generation can be viewed as designing a joint distribution over environments, tasks, and interaction realizations. The authors categorize generation paradigms into two main folds, forward generation and reverse generation.

Forward generation follows the natural dependency of interaction. It constructs an environment, generates grounded tasks based on that environment, and then obtains trajectories. This approach offers strong grounding but risks cascading dependence if the environment is narrow or unreliable. Reverse generation alters the starting anchor. Task-first pipelines specify a target capability and construct the necessary environment and interaction. Trajectory-first methods explore an environment or mine a workflow before writing the user-facing task. Structure-first generation creates an intermediate object, such as a tool graph or blueprint, to stabilize dependencies before full realization.

To shape the accepted distribution of generated data, the framework introduces the ACE objective. This objective treats Accuracy as an admission condition, while Complexity and Diversity shape the utility of valid data. The objective is formulated as:

$$\max_{\phi} \mathbb{E}_{\mathcal{B} \sim p_{\phi}} \left[ \lambda_{C} \frac{1}{|\mathcal{B}_{A}|} \sum_{d \in \mathcal{B}_{A}} g_{z}(C_{z}(d)) + \lambda_{D} D(\mathcal{B}_{A}) \right] \quad \text{s.t.} \quad \operatorname{Pr}_{d \sim p_{\phi}}[A(d) = 1] \geq \alpha$$ϕmax​EB∼pϕ​​[λC​∣BA​∣1​d∈BA​∑​gz​(Cz​(d))+λD​D(BA​)]s.t.Prd∼pϕ​​[A(d)=1]≥α

where $A(d)$A(d) is the validity decision, $\alpha$α is the required acceptance level, $C_z(d)$Cz​(d) is the difficulty of an instance under a specific execution configuration, and $D(\mathcal{B}_A)$D(BA​) is the batch-level diversity. This formulation ensures that accuracy establishes the feasible set, while complexity and diversity determine which valid data are most useful for training.

Accuracy is the prerequisite of the ACE objective. A sample must have an internally consistent environment, a feasible task, a valid interaction trajectory, and a correct verifier. The authors summarize recurring mechanisms for ensuring accuracy throughout the data generation pipeline.

The process begins with layered rule, model, and human-based checking to detect obvious errors through post-hoc filtering and turn-level review. Next, constraint-grounded construction prevents invalid construction by validating blueprints, graphs, and states before generation. Execution and state-based verification grounds the data in real execution, checking state transitions, outcomes, and executable objectives. Finally, feedback-based repair and selective admission allow the pipeline to revise, retry, and regenerate based on failure signals, admitting only the best candidates.

Complexity determines whether accurate data provides an informative learning signal. It is defined as model-relative and learner-dependent.

The learning utility is maximized within a learnable band between tasks that are too easy and redundant, and tasks that are too difficult and beyond the frontier. As the learner improves, this useful band shifts. The authors propose several mechanisms to construct and calibrate complexity. Structural specification and composition define dependency structures like subgoal graphs before realization. Task and information control adjust what must be achieved or what information is initially available, creating clarification and retrieval burdens. Environment and interaction design change the decision process through typed tool dependencies and persistent databases. Completion and feedback design control complexity through the definition of success, using path-centric rewards and step-level checks. Furthermore, failure-driven and model-aware calibration use solver behavior to retain candidates near a target success band, while bidirectional calibration and scaffolding simplify or strengthen candidates as needed.

Diversity concerns the breadth of accurate and appropriately complex agentic data, focusing on behaviorally distinct environment-task-interaction relations rather than surface variation.

To expand diversity, pipelines employ several mechanisms. Source and support expansion adds new environments, tools, and modalities. Compositional recombination creates new relations among existing components, such as multi-tool dependencies and cross-application workflows. Exploration and experience-first discovery traverse an environment to derive tasks from reachable behavior. Perturbation and counterfactual variation change selected factors while holding others fixed to expose specific shifts. Finally, coverage-guided balancing and adaptation target under-represented factor combinations and allocate generation toward named transfer gaps. The meaningful factors for diversity differ by domain, ranging from tool dependencies and state effects in digital agents to partner policies and private information in social agents, and materials or hypotheses in scientific domains.

Experiment

The paper evaluates agentic data generation pipelines across multiple dimensions of accuracy, complexity, and diversity, drawing on a broad survey of experimental evidence. Key qualitative findings indicate that accurate data requires verifying environments, tasks, trajectories, and verifiers together, with execution-based grounding providing stronger signals than plausibility checks alone, though it must be combined with semantic review to avoid loopholes. Complexity is only meaningful after validity is established, and structural proxies like horizon or tool count do not reliably predict behavioral difficulty; instead, frontier-oriented generation must balance difficulty with learnability and avoid curriculum drift. Diversity measurement cannot rely on a single metric: it demands factor coverage, behavioral non-redundancy, and evidence of transfer to held-out settings, while also accounting for the trade-offs with accuracy, model-aware complexity, and the sustainability of coverage over time.

Forward agentic data generation follows the environment-to-task-to-trajectory dependency, with environments drawn from real or curated sources, LLM synthesis, or programmatic construction. Each source trades off realism, scalability, controllability, and verification cost, shaping the diversity and reliability of the resulting trajectories. Hybrid approaches attempt to balance these factors, but diversity claims remain conditional on environment fidelity and verifier coverage. Real API collections and repositories provide authentic dynamics but inherit unstable dependencies, uneven documentation, and licensing constraints. LLM-synthesized environments scale easily and allow controlled variation, yet generated interfaces can be underspecified or detached from plausible workflows. Programmatic environments enable stateful interaction, controlled resets, and reliable verification at a higher engineering and maintenance cost. Expanding diversity through broader recombination increases the risk of incompatible tools and unreachable goals, requiring execution-based checks and verifier support. Diversity should be reported within difficulty bands to prevent models from losing foundational skills while chasing frontier capabilities. Hybrid designs that combine real specifications with executable replicas offer a middle ground, but their diversity claims must be tested on held-out real environments.

Reverse agentic data generation methods differ by which artifact anchors the pipeline: task-first approaches design target capabilities then synthesize environments and trajectories, while trajectory-first approaches mine interaction paths to derive tasks. These paradigms offer complementary control over diversity, executability, and alignment with real-world dynamics, but each introduces trade-offs such as bounded exploration, verifier mismatch, and mixture interference that require careful balancing. Task-first generation composes capability targets or multi-turn tasks before building tools and environments, enabling precise control over the skills being taught. Trajectory-first methods reverse interactions into tasks, grounding instructions in observed behavior to improve executability but limiting diversity to what exploration discovers. Scaling reverse generation can conflate support expansion with denser repetition, so coverage-controlled comparisons are needed to verify genuine diversity gains. Broader recombination increases the risk of incompatible tools and unreachable goals, requiring validity checks to expand alongside diversity to avoid discarding legitimate strategies. Real systems offer authentic dynamics but are costly and drift, while learned simulators scale cheaply but may invent transitions, making hybrid designs a practical but fidelity-dependent compromise. Mixing domains and generators introduces format and reward interference; standardization helps but can erase domain semantics, so sustainable diversity requires provenance tracking and periodic repair.

Representative domain-specific diversity methods center on tool use and search. Tool-use methods diversify through API coverage, verifiable function-calling execution, tool-pool and action-representation variation, or coherent multi-turn tool dependencies, while the search method diversifies through real-world research trajectories and evidence integration. The surrounding analysis indicates that these diversity gains are meaningful only when paired with validity checks, difficulty-aware allocation, and ongoing adaptation to shifting tools and environments. Tool-use methods in the representative set emphasize distinct diversity mechanisms, from large-scale real-world API coverage to verifiable execution and structured multi-turn dependency synthesis. The search-oriented method focuses on varied real-world research trajectories, retrieval behavior, and evidence integration rather than fixed tool-call patterns. The accompanying analysis cautions that broader diversity can introduce incompatibilities and requires verifier coverage, difficulty-band reporting, and periodic support repair.

Domain-specific diversity differs by agent type. Embodied generation centers on procedural or language-guided creation of scenes, objects, layouts, and physical interaction conditions, while social generation centers on open-ended variation in roles, goals, and partner behavior. Broader evidence shows tool-use, web or GUI, and coding agents diversify executable states, interfaces, workflows, and repository or trajectory coverage rather than treating nominal tool or page counts as sufficient. Embodied methods emphasize procedural scenes, objects, layouts, everyday simulation, and simulation-ready 3D worlds with physical interaction conditions. Social agent generation uses open-ended interactions to vary roles, goals, and partner behavior. Tool-use, web, GUI, and coding evidence indicates diversity claims depend on executable effects, visual versus behavioral variation, repository and trajectory coverage, and verifier or simulator fidelity.

The evaluation compares forward and reverse agentic data generation paradigms, examining how environment sources (real, LLM-synthesized, programmatic) trade off realism, scalability, and verification cost, and how task-first versus trajectory-first pipelines control skill coverage and executability. Domain-specific methods for tool use, search, embodied, and social agents show that diversity gains depend on validity checks, difficulty-aware allocation, and adaptation to shifting tools or environments, rather than on nominal API or page counts. Overall, the findings emphasize that expanding diversity risks incompatibility and requires execution-based verifiers, provenance tracking, and testing on held-out real environments to ensure that models retain foundational skills while pursuing frontier capabilities.