One-sentence Summary

Scale AI researchers propose an interaction-centric taxonomy that localizes agent failures to specific component interactions, assigns 41 failure modes to component edges and fault sides to directly guide repair, and, when evaluated across architectures, yields independent judges recovering human labels with Cohen’s $\kappa = 0.76$κ=0.76.

Key Contributions

• An interaction-centric taxonomy localizes agent failures to the component interaction where they originate and assigns a fault side (model, harness, environment, or grader), making repair targets directly actionable.
• The taxonomy organizes 41 failure modes from public benchmarks, system cards, and agent logs, and applies across coding, tool-use, and multi-agent architectures.
• Agent-as-a-judge experiments with four frontier models show that independent judges recover human failure labels well above chance, with the strongest reaching Cohen’s κ = 0.76, indicating the taxonomy captures shared structure rather than annotator-specific preferences.

Introduction

As LLMs are deployed in long-running autonomous settings, agents interact with users, tools, memory, and environments, creating a broad failure surface where identical symptoms can originate from different components. Prior taxonomies classify failures by outcome or internal module but cannot pinpoint which component is at fault, collapsing distinct causes and directing repairs toward the wrong part of the system. The authors introduce an interaction-centric taxonomy of 41 agent failure modes that locates each failure at the boundary between two components and assigns responsibility to the fault side, enabling targeted interventions such as model post-training or harness engineering. They validate the taxonomy by showing that independent reasoning agents can consistently recover human-assigned labels, indicating that the structure captures a reproducible failure landscape.

Method

The authors leverage a component-based framework to model agent systems and categorize failures. They represent each failure as an interaction edge paired with a fault side. The edge identifies the two components involved in the interaction, while the fault side identifies the specific component responsible for the failure.

The components are grouped into three families: User, Harness, and Environment. The User family includes the Owner, Grader, and Third party. The Harness family manages the model context, memory, tool access, and interactions with other models acting as peers or subagents. The Environment family covers the local execution setting and external services.

To localize a failure, the authors use a specific notation where an edge between two components is combined with the fault side. This is written as:

$$\underbrace{\mathrm{COMP}_{1} - \mathrm{COMP}_{2}}_{\text{edge}} \cdot \underbrace{\text{fault: SIDE}}_{\text{component at fault}}$$edgeCOMP1​−COMP2​​​⋅component at faultfault: SIDE​​

For example, a failure on the tool and model interaction where the model is at fault would be denoted with the edge TOOL-MODEL and fault assigned to the MODEL side.

When multiple errors contribute to a final outcome, the authors apply a root-cause principle. Starting from the observed system-level failure, they trace the causal chain backward to identify the earliest failure from which execution does not recover. Later errors are treated as consequences, and the taxonomy label is assigned to the interaction where this earliest unrecovered failure occurred.

The full taxonomy is organized hierarchically. The model sits at the root, branching into the component families, then specific components, and finally the failure modes associated with each interaction edge.

The taxonomy details specific failure modes for each family. For the User family, failures include Instruction-Following Failure or Sycophancy on the model side, and Instruction-Grader Mismatch on the owner side. For the Harness family, failures involve context management such as Goal Drift, memory operations like Missed Write or Pollution, and tool usage such as Tool Hallucination or Malformed Arguments. Multi-agent interactions are categorized by the role of the other model, distinguishing between peer and subagent failures such as Delegation Failure or Communication Failure. For the Environment family, failures include Service Failure or Stale State Delivery on the environment side, and Recovery Failure on the model side.

The authors developed this taxonomy iteratively by reviewing failures from public benchmarks and agent trajectories. Once the definitions stabilized, they froze the taxonomy to ensure consistent labeling. They applied the root-cause principle to assign labels, ensuring that each failure is attributed to the correct interaction edge and fault side based on the available evidence.

Experiment

The agent-as-a-judge experiment evaluates whether four frontier models can consistently apply the failure taxonomy by reconstructing evidence from varied sources and classifying the earliest unrecovered failure. Judges show substantial agreement with human labels on interaction categories, though agreement drops for complete failure-mode labels due to heterogeneous evidence and root-cause attribution challenges. Selective voting improves precision at the cost of coverage, and the results indicate that the taxonomy captures shared structure, making fault localization practical for distinguishing model-side, harness-side, and environment-side failures.

The taxonomy models an agent system as a set of interacting components, each with a clear definition, enabling failure localization by specifying the edge between two components and the side at fault. This decomposition distinguishes failures that originate from the model, the harness, the environment, or the evaluation setup, so that interventions can target the correct component. The owner specifies the task and success criteria, while the grader independently evaluates the outcome, allowing failures in evaluation to be separated from failures in following instructions. Context is the information available to the model during the current interaction, whereas memory is a persistent store that survives across sessions, and the tool component provides the bidirectional interface for model actions and observations. In multi-agent settings, the peer role describes interactions where neither model directs the other, while the subagent role indicates the focal model acts as the orchestrator.

All tested judges achieve substantial agreement with human labels on the interaction category, with exact-match accuracy between 0.75 and 0.80. Performance drops for the finer-grained failure-mode task, where accuracy ranges from 0.62 to 0.72 because the label set is larger and depends on identifying the correct category first. GPT-5.5 leads on both tasks, while the Claude-Opus models show similar category scores but greater variation in failure-mode accuracy. GPT-5.5 reaches the highest category accuracy (0.80) and failure-mode accuracy (0.72) among the judges. All three Claude-Opus judges share the same category accuracy (0.75), yet their failure-mode accuracy spans from 0.62 (Opus-4.7) to 0.70 (Opus-4.6).

Agent judges predicted failure-mode labels from source material, either with or without the human-assigned interaction category. When given the gold category, Claude Opus models showed a marked increase in accuracy and F1, indicating that many failure-mode errors arise from misclassifying the category first. GPT-5.5 did not benefit from the gold category, performing equally well in both settings. For Claude Opus models, providing the correct interaction category yields a noticeable jump in failure-mode accuracy and F1 over predicting both category and mode from scratch. GPT-5.5 achieves the same accuracy whether or not the gold category is supplied, suggesting its failure-mode errors are not concentrated at the category stage. Among Opus variants, the largest accuracy improvement from the gold category appears in the 4.6 and 4.8 versions, while 4.7 shows a smaller gain.

Selective voting among four judges reveals a trade-off between coverage and precision. Requiring stronger agreement on the interaction category increases category precision from 0.78 at full coverage to 0.96 at 68% coverage, while failure mode precision rises from 0.70 to 0.89. However, recall declines and the ensemble abstains on a growing share of examples, keeping F1 scores relatively stable. Moving from two to three agreeing judges lifts category precision from 0.78 to 0.83 while maintaining 90% coverage. Unanimity yields the highest category precision (0.96) and failure mode precision (0.89) but covers only 68% of examples.

The evaluation uses a component-based taxonomy to classify agent system failures into interaction categories and finer-grained failure modes. LLM judges achieve substantial agreement with humans on the interaction category task, but accuracy drops for the more granular failure-mode task, where misclassifying the category often leads to downstream errors. Providing the gold interaction category improves failure-mode accuracy for Claude models but not for GPT-5.5, indicating that GPT-5.5's errors are less concentrated at the category stage. Ensemble voting among judges reveals a trade-off between coverage and precision, with stricter agreement requirements yielding higher precision at the cost of reduced coverage.