One-sentence Summary

In a controlled study combining experiments and paired trajectory analysis across 8,135 trials, researchers from Princeton University, UC San Diego, Stanford University, and other institutions find that agent skills work mainly as procedural anchors that stabilize execution (65.7% of cases) rather than as explicit knowledge injection (4.5%), while retrieval precision drops from 29.6% to 3.3% as pools grow and skills fail under brittle assumptions.

Key Contributions

• Introduces a controlled contrastive evaluation methodology that combines quantitative experiments with paired trajectory analysis across 8,135 normalized trial records and derives a taxonomy of three high-level categories and twelve skill-use modes.
• Identifies procedural anchoring as the main benefit of skills, reporting a 6.06 point improvement over Workflow Memory in matched comparisons and attributing 65.7% of skill cases to procedural anchoring versus 4.5% to explicit knowledge injection.
• Quantifies retrieval and failure dynamics: actual-use precision drops from 29.6% to 3.3% as skill pools grow from 5 to 100, exact ground-truth invocation is neither sufficient nor necessary, and skills fail under brittle assumptions, incompatible contexts, or insufficient adaptation.

Introduction

Large language model agents that use tools increasingly rely on memory of prior executions, with skills emerging as compact procedural descriptions meant to compress noisy experience, standardize routines, and transfer knowledge across tasks. However, prior work evaluates skills mainly through aggregate task success, leaving unclear what changes in agent behavior, when skills help, and why they fail. The authors introduce a contrastive trajectory-analysis methodology that compares matched executions with and without skill access and analyzes divergence across representation, retrieval, and invocation. Their main contribution is a taxonomy of skill utility and failure plus evidence that skills work primarily as procedural anchors rather than factual knowledge, while failing when retrieved guidance is procedurally incompatible, weakly invoked, or insufficient for the execution bottleneck.

Method

The authors frame skill use as a controlled transformation from prior agent trajectories into reusable procedural knowledge. The core comparison is built around three execution arms: Raw, which receives no prior experience; Workflow Memory, which receives cleaned procedural traces from prior executions; and Skill, which receives a standardized SKILL.md artifact distilled from the same workflows. For each selected task, successful and failed raw trajectories are collected into a balanced trajectory pool. A fixed-budget composition grid then varies the evidence mix from success-only to failure-only, such as 5s0f through 0s5f. Workflow Memory and Skill are constructed from the same selected trajectories and evaluated on the same target tasks, holding the underlying experience constant while changing only its representation. To separate procedural content from explicit outcome signals, the authors also create standard and no-hint Skill variants. In the no-hint setting, success and failure identities are removed during skill construction, but the trajectories and execution protocol remain the same.

For cross-framework transfer, procedural artifacts are built from trajectories collected in the primary Codex setting and then evaluated in Gemini CLI with Gemini-3.1-Pro-Preview. The target-framework Raw baseline is used as the reference. This setup keeps the source experience fixed while the target agent differs in prompting style, tool-use interface, and execution behavior. It allows the authors to test whether distilled skills retain reusable procedural guidance more robustly than direct workflow traces when the agent framework changes.

The skill-retrieval study uses a controlled candidate-pool construction in which each pool contains the task’s ground-truth skill set plus distractors. Pool size ranges from 5 to 100, and distractors are sampled as random, semantically similar, or dissimilar skills. Three experimental arms measure complementary aspects of skill use: an embedding retriever that ranks skills by task description similarity, an explicit agent selection setting that chooses potentially useful skills without executing the task, and full-pool real execution in which the agent has access to the complete candidate pool. The offline diagnostics are run independently, and no selection output is transferred from either offline arm to the execution experiment. Ground-truth overlap of accessed skills and final verifier outcome are measured during execution.

To explain what changes when a skill is available, the authors build a contrastive trajectory-analysis pipeline. Heterogeneous benchmark outputs are normalized into a shared manifest of 8,135 trial records, of which 7,837 contain agent transcripts. An open-coding pass over 240 sampled trajectories produces 238 valid labels, which are merged into a 12-mode canonical taxonomy. The taxonomy construction is validated with independent human checks across 714 trajectory-label pairs, yielding 95.8% exact agreement and Cohen’s $\kappa = 0.952$κ=0.952. The main unit of analysis is a paired triple that compares the same task and setting under Raw, Workflow Memory, and Skill injection. The authors construct 528 such triples, giving 1,584 arm-level mode assignments. For each triple, an LLM judge assigns a taxonomy mode to each arm, records pairwise changes between arms, and identifies whether the injected artifact functions through procedural anchoring, knowledge injection, failure warning, no meaningful use, or counterproductive guidance. These fine-grained modes are grouped into coarse skill-use categories covering successful procedural anchoring, execution-layer and verification failures, and invocation or boundary failures.

Experiment

The experiments compare raw execution, workflow memory, and distilled skills across Codex and Gemini CLI on Terminal-Bench and SkillsBench, while also testing cross-framework transfer, outcome-label availability, and retrieval from controlled skill pools. Skills improve over raw traces mainly by acting as procedural anchors that stabilize execution and reduce environment or formatting failures, though they introduce new misapplication failures while workflow memory can cause process overload. Withholding outcome annotations matters primarily when failed trajectories are included, and retrieval studies show that semantically similar distractors degrade skill identification while downstream success remains comparatively flat, indicating that exact skill invocation is neither sufficient nor necessary. Overall, skill value depends on distillation, transfer, retrieval, and runtime adaptation rather than merely exposing agents to prior experience.

Skill injection consistently exceeds the raw baseline across all evaluated benchmarks and trajectory mixtures, while workflow memory exceeds raw only on SkillsBench and Terminal-Bench-Pro and falls below raw on Terminal-Bench-2. Workflow memory success generally declines as more failed source trajectories are included, with the lowest values under the mostly failed mixture. Skill injection also weakens at the extreme but can peak under mixed success-failure conditions rather than all-successful trajectories. Skill injection outperforms the raw baseline for every benchmark and mixture setting in this comparison. Workflow memory remains below the raw Terminal-Bench-2 baseline across all mixtures, while improving over raw on SkillsBench and Terminal-Bench-Pro. Including some failed source trajectories can improve skill success over all-successful trajectories, particularly on Terminal-Bench-2 and Terminal-Bench-Pro. Mostly failed source trajectories produce the weakest workflow results on Terminal-Bench-2 and Terminal-Bench-Pro, and reduce skill gains on Terminal-Bench-Pro.

Injected prior experience can be categorized as procedural guidance, missing domain knowledge, pitfall warnings, no meaningful use, or actively harmful use. As candidate skill pools grow, exact ground-truth skill-use precision drops sharply, but downstream task success stays relatively flat, so precise invocation is neither sufficient nor strictly necessary for success. Offline, similar distractors are the dominant challenge, and agents often consider the ground-truth skill along with distractors rather than overlooking it entirely. Mechanism labels distinguish beneficial procedural anchors and domain knowledge from null or counterproductive effects. Actual-use precision collapses as pool size increases, while downstream success remains comparatively stable across the same conditions. Semantic similarity among distractors is a stronger stressor than pool size alone, and high recall indicates the ground-truth skill is usually considered but not reliably selected.

Human validation covered two stages of taxonomy construction: confirming that raw labels were grounded in agent trajectories and independently mapping those labels to canonical modes. All sampled raw labels were confirmed as grounded in their supporting trajectories. Independent human and LLM assignments showed near-perfect agreement, supporting the stability of the taxonomy beyond a single LLM. Across 714 trajectory-label checks, every raw label was confirmed as grounded in recorded agent behavior. Independent human and LLM taxonomy mapping achieved 95.8% exact agreement and Cohen's kappa of 0.952.

Retrieval precision declines as candidate skill pools grow, with semantically similar distractors creating the largest drop in offline identification. Execution-time skill-use precision collapses to very low levels at larger pool sizes, while downstream task success remains relatively stable. This suggests agents often include or inspect the correct skill among distractors but do not reliably restrict use to the annotated ground-truth skill. Offline retrieval and explicit agent selection are substantially less precise for similar distractor pools than for random or dissimilar pools, and precision declines further as pool size increases. During execution, actual skill-use precision falls sharply with larger pools while task success stays comparatively flat, showing that exact skill-use matching and task completion are not tightly coupled.

These experiments compare skill injection and workflow memory against raw baselines across benchmarks and trajectory mixtures, examine how injected experience is used, stress-test retrieval and skill use with larger candidate pools and distractors, and validate the taxonomy through human review. Skill injection consistently improves over the raw baseline, while workflow memory helps only on SkillsBench and Terminal-Bench-Pro and weakens under mostly failed trajectories; injected experience can serve as procedural guidance or missing domain knowledge but can also be null or harmful, and larger and semantically similar distractor pools sharply reduce exact skill-use precision while downstream task success remains relatively stable. Human validation confirms that all sampled raw labels were grounded in agent behavior, with near-perfect agreement between independent human and LLM taxonomy mappings.