One-sentence Summary

Researchers from TraceLite AI, Institute of Automation CAS, et al. introduce TraceTrial-Bench, a benchmark that decomposes agent harness reliability into seven components with 95 mechanism-oriented tasks and includes Trajectory Tracing, Multi-source Evidence Acquisition, and Failure Attribution modules; across 4 harnesses, 9 models, and 3,420 runs, it reports macro-averaged harness scores ranging from 59.8% to 72.0%, with 31.1% of runs achieving task success despite failing at least one harness criterion.

Key Contributions

• TraceTrial-Bench is a benchmark that organizes agent harness reliability into seven components covering context, execution, governance, lifecycle, observability, tools, and verification, and provides 95 expert-level, mechanism-oriented tasks with executable or state-based ground truth to support semantically equivalent comparisons while preserving native harness behavior.
• The benchmark includes a Trajectory Tracing module, a two-layer evaluation framework that separately assesses agent performance and harness reliability, a Multi-source Evidence Acquisition module that draws evidence from the agent context, harness, and execution environment, and a Failure Attribution module that uses explicit mechanism chains for run-level attribution.
• Across 4 harnesses, 9 models, and 95 tasks (3,420 runs), macro-averaged harness scores range from 59.8% to 72.0%, and 31.1% of runs achieve task success despite failing at least one harness criterion; the released tasks, configurations, traces, and verdicts support auditable, reproducible comparison and failure diagnosis.

Introduction

Large language models have become the foundation of AI agents that carry out multi-step tasks by using tools, executing commands, modifying files, and interacting with external environments. Because agent behavior also depends on system instructions, context management, available tools, execution environments, persistent state, lifecycle controls, and safety mechanisms, the agent harness is a reliability-critical component. Most existing benchmarks evaluate only final answers, test results, or environment states, so they capture the combined effect of the model and harness but cannot reveal whether specific harness mechanisms failed; a task can succeed despite lost context or lifecycle violations, and fail despite faithful harness execution. The authors introduce TraceTrial-Bench, a mechanism-oriented benchmark that treats the native agent harness as the system under test. It uses auditable execution traces and external environment state across 95 tasks covering seven reliability components, with a two-layer evaluation framework that separates agent task performance from harness reliability and supports failure attribution through explicit mechanism chains.

Dataset

The authors describe TraceTrial-Bench as a formal evaluation set built from a 95-task population plus normalized trace archives.

Dataset composition and sources

• The task population is generated from taxonomy subdimensions. Each task identifies a native mechanism, the property under test, the model-visible consequence, and an external truth anchor.
• The benchmark contains 46 single-turn tasks, 45 fixed multi-turn tasks, and 4 simulator-driven tasks.
• Trace evidence is collected from four native harness adapters: Codex, Claude Code, OpenClaw, and DeepSeek Harness.

Task details and filtering rules

• Each task package fixes the prompt or turn sequence, initial workspace, allowed tools, budgets, expected outputs, deterministic verifier, two score sheets, and optional simulator or execution/security probes.
• Review filtering checks prompt solvability, mechanism reachability within budget, verifier agreement with task truth, hidden leakage, and equivalent domain semantics across eligible harnesses.
• Private simulator state and persisted baseline snapshots are kept outside the agent writable workspace.
• Task triggers include oversized outputs requiring recovery, nonzero exits paired with success-shaped artifacts, pending jobs, facts crossing session boundaries, and private values available only through an interaction channel.

Trace processing and metadata construction

• Codex trace acquisition reads its JSON event stream. Claude Code combines stream-json output with the session record. OpenClaw reads local or gateway-backed JSON output with session JSONL. DeepSeek consumes its persistent JSON-RPC notification stream.
• The archival policy stores a redacted copy and digest when native archives are available. Records that cannot be normalized are kept as provenance-tagged raw stdout or stderr events.
• Events are normalized into a JSON array with a contiguous seq index and a common envelope containing schema_version, seq, timestamp, run_id, and event_type.
• Tool calls use tool_call for the requested action and tool_result for the correlated return. The return can include typed error status, exit code, duration, truncation status, and omitted size. raw_event retains the parsed source record.
• Tool results are linked to calls only when the harness native call identifier is available. Unmatched, delayed, and duplicate records remain explicitly unlinked.
• Multi-agent lineage records principal and parent-principal identifiers, while native session and turn identifiers preserve harness boundaries.
• Provider-level tool declarations are extracted transiently from provider requests. They store names, descriptions, input schemas, request order, path, upstream status, timing, and usage, but exclude full request bodies and authorization material. A declaration is authoritative only for a successful request with a present, structurally valid, and complete declaration channel.
• No cropping strategy is described. Processing focuses on redaction, normalization, correlation, and source retention.

Use in evaluation

• The 95-task population is used as the formal evaluation set, not as training data. The text does not describe a training split or mixture ratios.
• Each task pairs Harness-layer criteria with Agent-layer deliverables and uses deterministic verifiers plus two score sheets.
• Trace archives support harness-fidelity evaluation by keeping multiple observations of the same action, such as an external process timeout that may be omitted from the model-facing result.

Method

The authors introduce TraceTrial-Bench, a comprehensive evaluation framework designed to assess agent harnesses as complete systems rather than transparent wrappers around models. The benchmark addresses three core questions for every admitted run: whether the agent produced the required artifacts and actions, whether the native harness preserved necessary information and execution semantics, and whether observed agent-side failures align with predeclared propagation mechanisms from harness-side criterion failures.

The overall architecture of the system is illustrated below.

The system under test is the native harness, which manages model interactions, tool coordination, lifecycle behavior, and permissions. The model, task, environment, and resource budgets serve as controlled inputs, while the evaluator and capture machinery function as the measurement apparatus. The pipeline operates through a sequence of collection, normalization, evidence selection, and evaluation steps, ensuring that apparatus failures are not counted as harness defects.

The system boundary and implemented dataflow are shown in the following diagram.

To structure the evaluation, the authors organize harness reliability into seven top-level dimensions: Context, Execution, Governance, Lifecycle, Observability, Tools, and Verification. These dimensions are defined by the harness obligation under test. The distribution of mechanism-level reliability properties across these components and their task coverage in the evaluation population is depicted below.

A critical component of the methodology is Trajectory Tracing and Multi-source Evidence Acquisition. Since native harnesses expose different observability interfaces, trace acquisition is implemented through source-specific adapters that map native observations to a common event schema. This allows for cross-boundary reconstruction of tool actions, distinguishing between the model-facing result, native records, external execution state, and normalized events. Retaining these separate observations enables precise harness-fidelity criteria.

The cross-boundary reconstruction of a semantic action is detailed in the figure below.

The evaluation process follows a strict two-layer protocol. A run is scoreable only after passing apparatus admission checks, which verify run identity, lifecycle records, archive integrity, and probe completeness. Once admitted, the system applies deterministic rules before proceeding to semantic judging. The Agent layer is evaluated first using changed artifacts as primary evidence. Only after the Agent-layer result is fixed are the Harness rubrics evaluated from trajectory and probe evidence. The final score combines the pass rates of both layers:

where $p_H$pH​ and $p_A$pA​ represent the number of passed items in the Harness and Agent layers, respectively, and $n_H$nH​ and $n_A$nA​ are the total finalized items in each layer.

The admission and layered-evaluation order is summarized below.

Finally, the framework includes a failure attribution mechanism that separates harness-side failures from agent-side failures. It identifies candidate mechanism links where a harness criterion failure and an agent criterion failure match predeclared endpoints and activation conditions. This allows the system to distinguish between independent failures and propagated failures, providing a rigorous basis for evidence-based failure analysis.

Experiment

The evaluation compares four agent harnesses across 95 tasks and nine model configurations using a two-layer scoring protocol that separates Agent task success from Harness-criterion satisfaction, with all runs archived under a reproducible full-grid protocol. The experiments validate that outcome quality and execution-layer reliability are empirically distinct: different harnesses lead the two aggregate measures, and overall task success does not imply that all harness criteria passed. Results further show that harness reliability differences depend on both the model and the taxonomy dimension, with mechanisms such as network restriction and tool exposure exhibiting notable shortfalls even when agent-level outcomes are strong. Failure co-occurrence is broader than confirmed candidate-link activation, indicating that the benchmark reveals diagnostic harness-level issues beyond superficial task failures.

The taxonomy spans seven reliability dimensions with uneven task coverage, led by Execution and Lifecycle and much lighter coverage in Observability and Verification. It organizes 40 mechanism-level reliability properties across Harness components with equal weight assigned to each property. Execution has the largest task share among dimensions, followed by Lifecycle and Tools. Observability and Verification are the least represented dimensions, together covering fewer than one in ten tasks. Conditional activation is lower in Execution and Governance than in Tools and Observability.

The evaluation design fixes task conditions before execution and separates agent deliverables, harness obligations, and independent truth anchors, allowing outcome quality and harness-criterion satisfaction to be measured as distinct outcomes. Across the full grid, agent success is much more common than passing every item in both layers, and different harnesses lead the two aggregate measures. Failure co-occurrence is broader than candidate-link activation, with many joint failures lacking stored links and many agent-failure runs retaining unreached failed criteria. Agent success reaches 39.5%, while only 8.4% of runs pass every item in both layers. Fewer than half of jointly failing runs contain a stored candidate link, and most agent-failure runs retain at least one failed criterion not reached by any link. The H-only state captures operational absorption, where measured harness shortfalls coexist with a fully passing agent outcome.

The tracer normalizes observed events into five canonical families: lifecycle, model surface, action/observation, stateful control, and accounting/archive. Optional fields appear only when native sources support them. In the analyzed runs, joint failure is the largest state, Harness and Agent verdicts frequently disagree, and failure co-occurrence is broader than stored candidate-link activation. The tracer separates normalized content into lifecycle, model surface, action/observation, stateful control, and accounting/archive families, with optional fields emitted only when native sources support them. Joint failure is the largest state, but Harness and Agent verdicts frequently diverge; failure co-occurrence exceeds stored candidate-link activation, and conditional activation is lower in Execution and Governance than in Tools and Observability.

The implemented evidence architecture separates archival evidence from criterion evidence and assigns each source a specific epistemic role. External state sources establish what occurred, model-facing records establish what was delivered, and correlated observations support fidelity comparisons across the two boundaries. Each criterion receives a bounded, auditable evidence packet with a deterministic and inspectable retrieval contract. External state sources establish what occurred, model-facing records establish what was delivered, and correlated observations support fidelity comparisons across these boundaries. Evidence packets bound rather than replace the full archive, returning short strings directly and retaining fixed boundary segments or keyword-centered windows for longer strings while reporting scanned, matched, and returned counts.

The judge resolution rules separate direct pass/fail states from Harness-only states that distinguish unexercised paths and genuine harness malfunctions. Unexercised Harness criteria are resolved from the Agent outcome, while malfunctions are finalized as Harness failures independent of Agent performance. Across runs, Harness and Agent verdicts frequently diverge, and joint failure is common but only partially overlaps with activated candidate links. Unsupported evidence finalizes as a fail and supported evidence finalizes as a pass for both Agent and Harness criteria. Not-exercised and malfunction states apply only to Harness rubric criteria and follow different finalization rules. A not-exercised Harness criterion passes only when all Agent criteria pass, whereas a malfunction always becomes a Harness fail. Joint failure is the largest observed state, but fewer than half of jointly failing runs contain a stored candidate link. Most Agent-failure runs retain at least one failed criterion not reached by any link, and fully passing Agent outcomes can coexist with Harness shortfalls.

The experiments evaluate a multi-dimensional reliability benchmark with seven dimensions, where Execution and Lifecycle dominate task coverage while Observability and Verification are rarely represented. The evaluation separates agent deliverables, harness obligations, and independent truth anchors, revealing that agent success is much more common than passing every agent and harness criterion, and that joint failures often lack stored candidate links. A tracer normalizes observed events into five canonical families, an evidence architecture separates archival from criterion evidence with bounded audit packets, and judge rules distinguish unexercised harness paths from genuine harness malfunctions. Overall, agent and harness verdicts frequently diverge, and fully passing agent outcomes can coexist with harness shortfalls.