One-sentence Summary

Researchers from Tencent HY LLM Frontier, Indiana University, and other institutions propose the Harness Handbook—a behavior-centric representation automatically synthesized from agent harnesses via static analysis and LLM-assisted structuring—and Behavior-Guided Progressive Disclosure, which guides coding agents to relevant code locations, significantly improving behavior localization and edit-plan quality over prior code-search methods, especially for modifications involving scattered implementation sites, rarely executed code paths, and cross-module interactions.

Key Contributions

• Harness Handbook, a behavior-centric representation, is automatically synthesized from a harness codebase through static analysis and LLM-assisted behavioral structuring, linking behavioral requirements to their scattered source implementations.
• Behavior-Guided Progressive Disclosure (BGPD) guides coding agents from high-level behavior descriptions to relevant code locations and verifies candidate locations against the current source, enabling targeted behavior localization before repository exploration.
• Experiments on Codex and Terminus-2 show that handbook-assisted planning improves plan quality (overall win rates +10.0 and +18.9 percentage points) and behavior localization (all 24 file- and symbol-level Recall, Precision, and F1 comparisons improve), while reducing planner token usage by 12.7% and 8.6%, with the largest gains for modifications involving scattered implementation sites, rarely executed code paths, and cross-module interactions.

Introduction

In modern LLM-based agentic systems, a software layer called the harness coordinates prompting, state management, tool invocation, and execution flow. As models, APIs, and requirements change, the harness must be repeatedly modified, making harness evolution a central and recurring engineering challenge. Any modification request first requires finding every source-code location that implements the described behavior, a task the authors call behavior localization.

Existing repository-exploration tools such as code search, maps, and summarization organize information around files, functions, and modules. They help identify individual pieces of relevant code but do not show how those pieces collectively produce a system behavior or guarantee that all affected locations have been found. Developers and coding agents are left to manually connect scattered implementation sites and infer how a high-level behavioral request maps to the underlying source code, which is costly and error-prone in large, production-scale harnesses.

The authors address this gap with Harness Handbook, a behavior-centric representation that directly links what the harness does to where that behavior is implemented in the codebase. It is constructed automatically using static program analysis and LLM-assisted behavioral structuring. The handbook reorients repository knowledge around runtime behaviors rather than static file structure, enabling a Behavior-Guided Progressive Disclosure workflow that leads coding agents from high-level behavior descriptions to precise implementation details. On realistic modification requests from two open-source harnesses, Handbook-assisted planning improves both behavior localization and edit-plan quality while using fewer planner tokens, demonstrating that making the behavior-to-code connection explicit can make harness evolution more accurate and efficient.

Dataset

The authors construct a handbook dataset from a software repository $R$R. The handbook is a structured, traceable documentation tree that links source code to execution stages, used for source localization and resynchronization in downstream tasks.

• Composition and sources

  • Built entirely from a single repository $R$R, with language-specific parsing.
  • Final output: a three-level hierarchy (L1 stages, L2 groups, L3 source units) plus a cross-stage state-register view.
  • The L3 leaf granularity $g$g is fixed as either function-as-leaf (each L3 entry covers a whole function or contiguous regions) or file-as-leaf (each L3 entry represents a file).

• Processing flow

  • Phase I – Static fact extraction: Deterministic; no LLM calls. Extracts functions, named boundaries, source locations, signatures, and call edges. Only calls resolving to internal functions or named boundaries are kept; unresolved calls are logged.
  • Phase II – Behavioral organization: LLM-driven proposal-and-review steps assign source units to execution stages.
    • Function-as-leaf: uses a provided seed skeleton; functions can be sliced into regions for multi-role behavior.
    • File-as-leaf: infers the stage skeleton from scanned file summaries and the program graph, with optional iterative refinement. Uncovered files and issues are explicitly recorded.
  • Phase III – Hierarchical synthesis and packaging: Converts the stage skeleton and assignments into the L1–L3 document tree and state-register view. Every L3 entry is linked to a statically identified source location and validated against the current repository. The pipeline then packages the structured data for source localization and future resynchronization.

• Usage in the model The handbook serves as a code-linked representation that ties execution stages to precise source locations. It is used to support downstream source localization tasks and to enable resynchronization when the repository evolves.

Method

The authors propose Harness Handbook, a framework designed to understand and modify an agent harness by organizing source code around runtime behavior. The system comprises three core components: a behavior-oriented representation, a construction pipeline to build this representation from a repository, and a modification workflow that uses the handbook to guide code changes and automatically resynchronizes it.

The Harness Handbook representation reorganizes repository information around behavior while preserving links to the source. It consists of an L1 to L3 document tree and a complementary state-register view. Readers navigate progressively from L1 (system overview) to L2 (component overview) and finally to L3 (unit deep dive), which links stages to source-grounded implementation entries. The state-register view records state relationships that cross stage boundaries. Two key rules govern this representation: progressive disclosure, ensuring readers access detail only when needed, and behavior-implementation alignment, ensuring every active L3 locator resolves to the current repository.

The construction pipeline builds the handbook from a repository using a fixed leaf mode $g \in \{function, file\}$g∈{function,file}, which determines the granularity of L3 entries. The pipeline operates in three distinct phases, as shown in the figure below:

Phase I, Static Fact Extraction, uses language-specific adapters to parse the repository and extract functions, boundaries, source locations, signatures, and call edges into a program graph. This phase is entirely deterministic and requires no LLM calls. Phase II, Behavioral Organization, maps source units to an execution-stage skeleton. In function-as-leaf mode, the pipeline proposes function-to-stage assignments and refines them through an iterative propose-and-review loop. In file-as-leaf mode, it summarizes files and infers the stage skeleton. Phase III, Hierarchical Synthesis and Packaging, converts the stage skeleton and source organization into the L1 to L3 document tree and cross-stage state-register view, validating each L3 entry against the repository to ensure traceability.

Once constructed, the handbook guides repository modifications through a four-step workflow. First, Behavior-Guided Progressive Disclosure localizes the requested behavior by navigating the handbook from coarse execution stages down to specific L3 entries, expanding candidates along call relations, and verifying them against the current repository to produce source-grounded evidence. Second, a planner converts this evidence into an edit plan and action declarations. Third, an executor applies the plan to the repository. Finally, any non-empty diff triggers handbook resynchronization. The resynchronization process reparses the changed source, aligns old and new versions to identify modified units, and updates only the affected parts of the handbook, keeping model calls limited to specific semantic steps while handling unparseable content conservatively.

Experiment

The evaluation compares a handbook-assisted planner against a baseline on two open-source agent harnesses using 30 behavior-driven modification requests, with plan quality judged by three independent models. Handbook guidance consistently improves plan quality, localization accuracy, and scope control while reducing planner token use, enables a weaker planner to better align with reference plans from stronger models, and these gains persist across diverse request types and labeled difficulty levels.

Providing handbook guidance to a weaker planner consistently boosts its localization agreement with independent reference plans. Across two codebases, two judge models, and both file and symbol granularities, all F1 scores are higher, with gains between 5 and 19 points, driven by simultaneous increases in recall and precision. The guidance also cuts the rate of complete misses by up to 26 points, showing that improvements include eliminating truly wrong plans. Handbook assistance lifts F1 scores by 5.0 to 18.8 points across all harnesses, reference models, and granularities, with recall and precision rising together. The share of requests with zero overlap against the reference plan drops by as much as 25.9 points with handbook guidance, measuring a reduction in complete localization failures.

Providing handbook guidance to a weaker planner consistently improves its localization agreement with independent reference plans across diverse evaluation settings (two codebases, two judge models, file and symbol granularities). F1 gains of 5–19 points arise from simultaneous increases in recall and precision, and the share of plans with zero overlap drops by up to 26 points, indicating fewer complete localization failures.