One-sentence Summary

Tencent HY LLM Frontier, Indiana University, and colleagues introduce the Harness Handbook, a behavior-centric representation automatically synthesized via static analysis and LLM structuring that maps agent harness behaviors to source code, and Behavior-Guided Progressive Disclosure, which guides coding agents from high-level behavior descriptions to relevant implementation details, thereby substantially improving behavior localization and edit-plan quality for complex modifications to AI agent harnesses, especially when changes involve scattered implementations or cross-module interactions.

Key Contributions

• The paper introduces Harness Handbook, a behavior-centric representation automatically synthesized from a harness codebase through static program analysis and LLM-assisted behavioral structuring, which links each system behavior to its distributed source implementations.
• It presents Behavior-Guided Progressive Disclosure (BGPD), a method that guides coding agents from high-level behavior descriptions to relevant implementation details and verifies candidate code locations against the current source.
• Evaluations on two open-source agent harnesses show that handbook-assisted planning improves behavior localization and edit-plan quality while using fewer planner tokens, with the largest gains for changes involving scattered implementation sites, rarely executed code paths, and cross-module interactions.

Introduction

The authors address the challenge of modifying production-scale agent harnesses, the software layer that coordinates prompt construction, tool invocation, state management, and execution flow in LLM-based agentic systems. As models and environments evolve, developers or coding agents must locate every implementation site responsible for a desired behavioral change, a task the authors term behavior localization. Prior repository representations, such as maps, code search, and summarization, organize code by files and functions but fail to show how scattered code pieces collectively produce a runtime behavior; coding agents still face iterative, error-prone exploration. To bridge this gap, the authors introduce Harness Handbook, a behavior-centric representation that explicitly links what the harness does to the source code that implements it, built automatically through static analysis and LLM-assisted structuring. They also propose Behavior-Guided Progressive Disclosure (BGPD), which guides agents from high-level behavior descriptions to concrete code locations. Evaluations on two open-source harnesses show that Handbook-assisted planning improves behavior localization and edit-plan quality while reducing planner token consumption.

Dataset

The authors construct a harness handbook from an existing code repository; this handbook serves as the structured knowledge source used by the model. Its composition and processing are as follows:

• Dataset composition and sources

• The sole source is a code repository R (C, C++, or similar) representing a harness or firmware library.

• The resulting handbook is a hierarchical document with three levels (L1–L3) and a cross‑stage state‑register overview.

• Two leaf granularities are available:

• function‑as‑leaf – each L3 entry covers a whole function or contiguous regions of a function.

• file‑as‑leaf – each L3 entry represents an entire source file.

• Key details for each subset/mode

• function‑as‑leaf mode:

• Used when a trusted seed skeleton that faithfully reflects execution stages exists and function‑level detail fits a budget.

• Seed skeleton provides initial stage and state‑register definitions.

• file‑as‑leaf mode:

• Used when no seed skeleton is available or function‑level organization would exceed the budget.

• The pipeline infers the stage skeleton from file summaries and program graph; an optional review loop refines the structure.

• Both modes produce the same L1–L3 representation and flag unresolved issues in a separate issues record Y.

• How the data is used

• The handbook is used as a model input for source localization and resynchronisation tasks; it provides a traceable mapping from behavioural stages back to specific source locations.

• The training/usage mixture is not a traditional data split but a single constructed handbook; the mode is selected once per project based on trustworthiness of the seed skeleton and budget.

• Processing and construction pipeline

1. Static Fact Extraction (Phase I)

• Language‑specific adapters parse the repository to extract functions, named boundaries, source locations, signatures, and call edges.
• A program graph G keeps only internal calls or calls to named boundaries; unresolved calls are logged, not guessed.
• This phase is deterministic and uses no language model calls.

2. Behavioral Organization (Phase II)

• Function‑as‑leaf: proposes function‑to‑stage assignments using source and call graph context; assignments are reviewed iteratively and can map a function to multiple stages or split it into contiguous regions.
• File‑as‑leaf: generates file cards, combines them with the program graph to infer stage skeleton S, organises files by stage, and optionally refines via proposal and review.

3. Hierarchical Synthesis and Packaging (Phase III)

• Converts the stage skeleton and organisation into the L1–L3 document tree and cross‑stage state‑register view.
• Each L3 entry is linked to a statically identified source location and validated against the current repository, ensuring traceability.
• The chapter view V is rendered and structured data is packaged for downstream source localisation and future resynchronisation.

Method

The authors introduce Harness Handbook, a system designed to understand and modify an agent harness by organizing source code around runtime behavior. The system comprises three core components: a behavior-oriented representation, a construction pipeline, and a modification workflow.

Harness Handbook Representation Source repositories indicate where code is stored but do not directly reveal how runtime behavior unfolds across files and stages. The Harness Handbook reorganizes this information around behavior while preserving links to the source. As shown in the framework diagram, the representation consists of an L1–L3 document tree $D$D and a complementary state-register view $Z$Z.

Readers typically start at L1 (system overview), which summarizes the architecture, execution model, major stages, and global data flow. They then move to L2 (component overview) for the responsibilities, inputs, outputs, dependencies, and local state of a selected stage. Finally, L3 (unit deep dive) links that stage to source-grounded implementation entries. The complementary view $Z$Z records state relationships that cross stage boundaries. Two rules maintain the representation's utility: progressive disclosure, where readers move from L1 to L3 only when more detail is required, and behavior-implementation alignment, ensuring every active L3 locator resolves to the current repository.

Construction Pipeline The construction pipeline builds the handbook from a repository $R$R using a fixed leaf mode $g \in \{function, file\}$g∈{function,file}, which determines the granularity of L3 entries. Once the leaf mode is chosen, construction proceeds in three phases.

Phase I, Static Fact Extraction, uses language-specific adapters to parse the repository and extract functions, named external boundaries, source locations, signatures, and call edges. This phase is deterministic and produces a program graph $G$G. Phase II, Behavioral Organization, organizes source units into the execution-stage skeleton $S$S. In function-as-leaf mode, the pipeline uses source and call-graph context to propose function-to-stage assignments, refined through iterative review. In file-as-leaf mode, the pipeline summarizes scanned files as cards and combines them with the program graph to infer the stage skeleton. Phase III, Hierarchical Synthesis and Packaging, converts the stage skeleton and source organization into the L1–L3 document tree and cross-stage state-register view. Each L3 entry is linked to a statically identified source location and validated against the current repository.

Handbook-Guided Modification and Resynchronization After construction, the handbook serves as a behavior-oriented guide for modifying the repository. Given a request $q$q, the workflow uses the handbook $H$H and repository $R$R.

The workflow begins with Behavior-Guided Progressive Disclosure (BGPD), which localizes the requested behavior through coarse-to-fine handbook navigation. BGPD identifies relevant execution stages using L1 and L2, follows the state-register view $Z$Z to include coupled stages, and selects relevant L3 entries. It then expands the candidate set along call relations and verifies these sites against the current repository to produce verified evidence $\widehat{E}_{q}$Eq​.

Next, a planner converts $\widehat{E}_{q}$Eq​ into an edit plan $P$P and action declarations $\Gamma$Γ. An executor applies $P$P to the repository, producing an updated repository $R'$R′ and a diff $\Delta$Δ.

Finally, any non-empty diff triggers handbook resynchronization. The procedure reparses the changed source, refreshes the program graph, and aligns old and new versions to identify added, removed, and modified units. If the stage skeleton remains valid, only affected entries are refreshed; otherwise, the construction algorithm is rerun. This ensures the handbook remains consistent and up-to-date with the repository.

Experiment

The experiments assess handbook-guided localization for edit planning on two open-source agent harnesses, comparing a baseline that explores code directly against an arm that follows a navigation policy based on the handbook. Handbook assistance consistently improves plan quality across judges and dimensions while reducing planner token usage. It enables a weaker planner to match the localization accuracy of stronger models, cutting complete misses, and these gains persist across request types and difficulty levels.

Handbook guidance consistently improves reference-plan localization metrics (Recall, Precision, F1) across two harnesses, two reference models, and both file and symbol granularities, with F1 gains reaching up to 18.8 points. Recall and precision rise together, indicating better focus rather than simply returning more candidate sites, and the share of requests with zero overlap (Wrong) never increases and falls by as much as 25.9 points, meaning fewer complete localization failures. On the Terminus-2 harness, handbook assistance brought file-level F1 to 84.7% and 89.3% against Opus 4.8 and GPT-5.5 references, respectively, and precision against GPT-5.5 reached 93.3% at both file and symbol granularity. The Wrong rate, which measures requests with no overlap to the reference, dropped by up to 25.9 percentage points; on the Codex harness against the Opus 4.8 reference, Wrong fell from 37.0% to 14.8%.

The evaluation measured the effect of supplementary handbook guidance on reference-plan localization using two harnesses, two reference models, and both file and symbol granularities. Guidance consistently raised recall and precision in tandem, lifting F1 scores by as much as 18.8 points and driving down the rate of completely missed localizations by up to 25.9 percentage points, indicating that the handbook narrows attention rather than simply surfacing more candidates. These gains held across all setups, demonstrating that the guidance robustly improves localization accuracy and reduces outright failures.