One-sentence Summary

Researchers from the University of Science and Technology of China, Shanghai AI Laboratory, and Fudan University propose FACET, a framework that preserves source intent and cross-artifact consistency by reconstructing agent skills into coherent scenarios, grounding the instruction, solution, and verifier in a realized container state, and applying execution-based validation with targeted repair to synthesize complex terminal tasks that provide data-efficient fine-tuning gains on Terminal-Bench 2.1.

Key Contributions

• FACET reconstructs related agent skills into coherent, information-rich terminal-task scenarios and realizes and repairs the execution environment before generating the final instruction, reference solution, and verifier.
• An executable-state-grounded construction paradigm uses a shared realized container state to coordinate the instruction, solution, and verifier as a single coherent task; execution-based validation and targeted repair then correct artifact-specific failures without regenerating already valid components.
• Sequential environment-grounded construction achieves the highest task yield and improves solution–verifier alignment over generating the verifier before the solution; fine-tuning Qwen3.5 models from 4B to 27B on FACET-generated successful trajectories improves Terminal-Bench 2.1 scores by 7.12, 8.24, and 6.75 absolute points, with the 27B model reaching 47.57 and approaching the 49.06 score of Qwen3.5-397B.

Introduction

The authors address the growing need for high-quality supervision data to train language agents that operate in terminal environments, where success demands correctly coordinating an instruction, an execution environment, a reference solution, and an executable verifier. Prior synthetic pipelines struggle with two key problems: rich source information is progressively lost across multi-stage generation, and task components can drift apart when they are not anchored in a shared, realized execution state. To overcome these issues, the authors introduce FACET, a framework that first reconstructs coherent user scenarios and cross-skill dependencies from a large collection of reusable agent skills, then builds and repairs the execution environment, and uses the realized container state as a common grounding interface before finalizing the instruction, solution, and verifier, followed by execution-based validation and targeted artifact repair.

Dataset

The authors construct an executable task dataset from public skill packages.

• Sources and initial filtering

  • Public skill packages are collected from OpenClaw, ClawHub, and GitHub.
  • The authors remove unsafe instructions, skills requiring private websites or private information, and skills depending on non-public resources.
  • They also discard unreadable, non-actionable, and duplicate records.

• Skill records and statistics

  • Each valid skill is normalized into a structured record containing description, required tools, inputs and outputs, procedural steps, and source provenance.
  • This yields more than 71K valid skills.
  • Retained skills are organized into 5 top-level and 34 fine-grained categories.
  • Final validated tasks are grouped into 9 task families.
  • The provided sections do not state the exact final number of validated tasks or downstream split sizes.

• Scenario extraction and repository construction

  • An extraction agent identifies possible application contexts, user goals, initial states, and desired final states for each skill.
  • Scenario hypotheses are embedded, and similar hypotheses from other skills are retrieved to identify related skill combinations.
  • Similar hypotheses are grouped into candidate scenario-skill pairs.
  • A model-based judge retains only candidates that are relevant, complementary, non-redundant, and executable as terminal workflows.
  • The accepted pairs form the repository used downstream.

• Metadata construction

  • Each accepted scenario-skill pair is reconstructed into five dimensions: goal, context, capability, state, and inputs/outputs plus tools.
  • These dimensions are integrated into a complete natural-language scenario, which serves as the shared semantic reference.
  • The authors then generate a solution reference and an instruction reference from this scenario and check that the two references are consistent.

• Executable task package construction

  • Final items are packaged in the Harbor format with environment/, solution/, tests/, instruction.md, and task.toml.
  • The environment agent first writes a manifest describing directories, files, services, and dependencies, then materializes it inside a restricted base image.
  • Retrieved resources are localized into the build context, making the environment self-contained with no network access needed during evaluation.
  • The authors may augment or perturb generated or collected data with extra records, metadata fields, distractor entries, and cross-file relations while preserving schema and task semantics.
  • Validation checks that the environment builds and initializes, the verifier does not pass in the initial state, the reference solution executes from a clean initial state, and the verifier passes on the final state.
  • Failed candidates are sent to targeted repair and re-evaluated from a clean state. Candidates still invalid after the repair budget are discarded.

• Usage and processing notes

  • The reconstructed scenario and aligned references are used as the shared specification for generating the instruction, solution, and verifier from the same realized container state.
  • No fixed-length cropping is described. Instead, the pipeline uses semantic filtering, scenario extraction, similarity grouping, judge-based retention, and agentic reconstruction to shape the dataset.
  • The provided material does not specify training, validation, or test split ratios or downstream mixture ratios.

Method

The authors structure the FACET pipeline as a three-stage generation process that transforms raw skill descriptions into fully validated, executable terminal task bundles. A task bundle $\mathcal{T} = (\mathcal{I}, \mathcal{E}, \mathcal{S}, \mathcal{V}, \mathcal{M})$T=(I,E,S,V,M) is composed of a user instruction, an environment specification, a reference solution, an executable verifier, and runtime metadata. The acceptance criterion for a synthesized task requires that the environment builds successfully, the verifier fails on the initial state, the reference solution executes without failure, and the verifier passes on the final solution state:

The pipeline enforces this logical check at the final validation step, and earlier stages are designed to maximize the likelihood that a candidate satisfies all four conditions.

The first stage, information source acquisition, collects publicly available skill packages from sources such as OpenClaw, ClawHub, and GitHub. A filtering step removes skills containing unsafe instructions, dependencies on private resources, or non-actionable content. Each retained skill is normalized into a structured record that captures its description, tool set, inputs/outputs, procedural steps, and provenance. An extraction agent then identifies potential application contexts, user goals, and desired final states for each skill, embedding these scenario hypotheses and retrieving similar hypotheses from other skills to detect meaningful combinations. A model-based judge evaluates candidate scenario–skill pairs $p_c = (c, X_c)$pc​=(c,Xc​) for relevance, complementarity, non-redundancy, and executability, keeping only those deemed valid:

The resulting scenario–skill repository $\mathcal{P}$P serves as the foundation for the next stage.

The second stage expands each scenario–skill pair into a richer compositional specification through agentic reconstruction and reference building. The pipeline first applies five progressive modules: skill analysis determines capabilities, preconditions, and effects; scenario exploration proposes concrete application settings where the skills contribute to a shared objective; association and filtering discards shallow side-by-side placements; evolution and recovery arranges capabilities into a coherent workflow and recovers cross-skill dependencies, intermediate artifacts, and state transitions; and information expansion enriches the workflow with concrete resources, formats, constraints, and success conditions. The resulting scenario is then described simultaneously along five dimensions:

These dimensions capture the user objective, application setting, role and inter-relationships of skills, initial/intermediate/final states, and all required files, schemas, tools, and services. A model integrates the five descriptions into a complete natural-language scenario $C$C. From this shared semantic reference, the system generates a solution reference $R_S$RS​ (setup actions, workflow, state transitions) and an instruction reference $R_I$RI​ (goal, inputs, outputs, constraints) with a consistency-alignment model ensuring that both references share the same initial state and target outcome:

The aligned references are passed to the final stage.

The third stage, executable-state-grounded task construction, converts the specification $Z = (C, R_S, R_I)$Z=(C,RS​,RI​) into an actual task bundle by first building and repairing the execution environment. An environment agent separates planning from materialization: it produces a manifest of directories, files, services, and dependencies, then materializes them inside a restricted base image using network access, shell and Python programs, and data augmentation with distractor records and cross-file relations to avoid trivial fixtures. The resulting image is built, and initialization checks are performed; failures such as compiler errors, missing packages, or service crashes trigger a targeted repair loop (up to three iterations) conditioned on both the failure trace and the original specification to prevent regressions.

After a successful build, the realized container state is exposed as a shared grounding interface. The instruction, solution, and verifier are generated sequentially, each with read access to the same concrete environment: the instruction is grounded in actually existing files and services; the solution is constructed while inspecting the environment directly; and the verifier is produced last, comparing initial and final states against the reference workflow, favoring behavioral and state-based checks over exact command matching.

Each candidate is packaged in the Harbor format and subjected to validation against the four acceptance conditions. When validation fails, a constrained router identifies the responsible component from execution traces and invokes only the corresponding repair procedure (environment, solution, or verifier repair), preserving valid parts and avoiding full regeneration. Task-level repair is limited to at most five iterations, and candidates still invalid after the budget are discarded. This three-stage pipeline, from skill acquisition through environment-constrained grounded generation and targeted repair, produces executable tasks with high structural consistency and verifiable state transitions.

Experiment

This set of experiments evaluates a terminal-agent training pipeline that uses Terminus-2 rollouts to fine-tune Qwen3.5 models at multiple scales and tests them on Terminal-Bench 2.1 with execution-based verification. The comparisons show that the FACET dataset has fewer but longer and more heavily checked trajectories than prior terminal datasets, and that even a small set of successful trajectories yields consistent fine-tuning improvements across model sizes. Task analysis indicates that many failures reflect unmet secondary or interdependent requirements despite substantial correct progress, while generation-scheme experiments confirm that producing solutions before verifiers improves task validity and cross-artifact alignment.

FACET offers long-horizon trajectories averaging 11.86 turns, comparable to TerminalWorld and longer than most other datasets, despite a modest training set of 1.2K trajectories. Its tasks carry the highest number of executable tests per task (22.77), enforcing stricter multi-requirement verification that leads to lower pass rates (P@1 27.00, P@3 35.00) compared to other datasets. The improvement from P@1 to P@3 indicates some failures are recoverable with retries, while the remaining gap highlights tasks that consistently expose requirement-satisfaction errors. FACET trajectories average 11.86 turns, similar to TerminalWorld (11.94) and longer than Tmax (11.14), Nemotron-Terminal (6.12), Terminal-Lego (5.77), and Endless-Terminals (4.53). FACET tasks include 22.77 executable tests on average, far exceeding the next highest dataset (Terminal-Lego at 16.60) and more than five times the lowest (Tmax at 3.29). The P@1 of 27.00 and P@3 of 35.00 are the lowest among all compared datasets, consistent with the larger number of independently checked requirements per task. The 8-point gain from P@1 to P@3 shows that repeated attempts recover some failures, while the remaining gap to higher-performing datasets indicates many tasks persistently expose requirement-satisfaction errors.

Fine-tuning on only 1.2K successful trajectories yields consistent Terminal-Bench 2.1 gains across 4B, 9B, and 27B models. The 9B model shows the largest absolute improvement, while the 4B model shows the largest relative gain, and the fine-tuned 27B model closes most of the gap to a much larger 397B model. These results indicate the supervision transfers across model scales rather than benefiting only one capacity regime. Training on 1.2K successful trajectories improves all evaluated scales on Terminal-Bench 2.1. The 9B model records the largest absolute gain, while the 4B model has the largest relative improvement of 40.5%. The fine-tuned 27B model lags a 397B model by only 1.49 points despite being roughly 15x smaller.

FACET is a challenging long-horizon benchmark with many executable tests per task, which enforces strict multi-requirement verification and results in low pass rates, though some failures can be recovered through retries. Fine-tuning on only 1.2K successful trajectories from FACET yields consistent performance gains across model scales, with the 4B model showing the largest relative improvement and the 27B model nearly matching a 397B model. These results demonstrate that the supervision signal transfers effectively across different model capacities.