One-sentence Summary

Researchers from the University of Illinois Urbana-Champaign, Meta, and Stanford University present a unified survey on 'code as agent harness,' structuring agent infrastructure into interface, mechanism, and scaling layers to enable executable, verifiable, and stateful AI systems for coding assistants, GUI/OS automation, embodied agents, and enterprise workflows.

Key Contributions

• The survey introduces the concept of "code as agent harness," which positions code as the operational substrate for agent reasoning, action, environment modeling, and execution-based verification.
• The survey is organized around three layers: harness interface (connecting agents to reasoning, action, and environment modeling), harness mechanisms (planning, memory, tool use, and feedback-driven control), and scaling (multi-agent coordination via shared code artifacts).
• The survey identifies open challenges such as multimodal verification, regression-free harness improvement, and safe long-horizon autonomy, and outlines a roadmap toward executable, inspectable, stateful, and governed agent systems.

Introduction

The authors survey a shift in large language model (LLM) agent design: code is no longer just a generated artifact but an executable, inspectable, and stateful medium through which agents reason, act, and coordinate. While prior surveys treat code as the end product of LLMs, this work highlights that the reliability of long-running agentic systems depends on the software harness that connects model outputs to tools, memory, execution, and feedback, and that agent-initiated code artifacts remain underexplored. The authors contribute a conceptual framing of “code as agent harness” and a three-layer taxonomy covering harness interfaces (code for reasoning, acting, and environment modeling), harness mechanisms (planning, memory, tool use, and feedback-driven control), and scaling the harness to multi-agent coordination, along with an analysis of open challenges in evaluation, verification, safety, and harness evolution.

Method

The authors frame the design of LLM-based agents around the principle that code serves as the central harness interface, transforming a stateless language model into a functional agent that can reason, act, and model its environment through executable, inspectable, and stateful artifacts. Rather than relying solely on natural language, the harness grounds the model’s outputs in external execution, persistent state, and verifiable feedback. Code, broadly defined to include programs, scripts, formal specifications, proof scripts, API schemas, tests, repositories, simulators, and execution traces, provides the medium through which the model’s intentions become operations with formally verifiable outcomes, intermediate computation is exposed as structured traces, and task progress is maintained in a persistent, modifiable form across steps.

The overall architecture is organized around three roles that code assumes within the agent loop: reasoning, acting, and environment modeling. Code for reasoning externalizes internal logic into verifiable computation. The model generates executable programs that external runtimes, interpreters, or symbolic solvers can execute and evaluate, separating high-level decomposition from low-level computation. This paradigm encompasses program-delegated reasoning, where the model produces code that interpreters execute to yield formally grounded outputs; formal verification and symbolic reasoning, where proof assistants or symbolic solvers check each derivation step; and iterative code-grounded reasoning, which closes the loop between generation, execution, and feedback through repeated generate–execute–verify–refine cycles, often using reinforcement learning with execution-based rewards.

Code for acting translates high-level intent into executable operations that interact with embodied, GUI, software, or tool-use environments. Here the harness must map abstract language outputs into grounded behaviors that respect the constraints of the target environment, including embodiment limits, interface APIs, and safety requirements. The interface is realized through grounded skill selection, where the agent selects from a library of executable capabilities under feasibility constraints; programmatic policy generation, in which the harness directly materializes control policies as code that specifies perception-conditioned branching, feedback loops, and API interaction; and lifelong code-based agents, where executable skills and interaction traces are accumulated as persistent memory, enabling the agent to evolve its capabilities over long-horizon deployment.

Code for environment modeling introduces executable programs as the environment interface itself, materializing world state, transition dynamics, and feedback signals through computational artifacts such as simulators, repositories, tests, execution traces, and state-transition programs. This allows the agent to explicitly store, inspect, execute, and modify environment state. The approach spans structured world representations, where environments are encoded as programmatic structures that the agent can manipulate; execution-trace world modeling, where runtime transitions themselves become the primary representation of environment behavior; and verifiable environment construction, where the harness synthesizes runnable environments with transition dynamics and verification oracles, turning the environment into a reproducible program world.

Building on this interface, the harness mechanisms layer provides the control infrastructure that makes code-harnessed agents reliable beyond a single generation step. Planning organizes long-horizon task execution by externalizing goals into decompositions, structural constraints, search trajectories, or workflow-level orchestration. It can take the form of linear decomposition, structure-grounded planning over dependency graphs or repository graphs, search-based planning that explores multiple candidate trajectories using Monte Carlo tree search or other inference-time compute allocation, and orchestration-based planning where the harness coordinates specialized roles, stages, and feedback loops through explicit contracts and runtime interpretation.

Memory and context engineering manage mutable state across long interactions. The harness maintains working memory for the current trajectory, semantic memory that retrieves repository evidence, experiential memory that captures reusable debugging or repair patterns across tasks, long-term memory that preserves validated knowledge in a compact form, and multi-agent memory that synchronizes shared state across roles. Context compaction and state offloading further separate decision-relevant active context from durable full-fidelity artifacts, making memory scalable and auditable.

Tool usage expands the agent’s action space while exposing governed interfaces to external systems. Tools are categorized by their primary harness function: function-oriented tools ground the agent in APIs and documentation; environment-interaction tools allow the agent to act inside repositories, terminals, and sandboxes; verification-driven tools provide deterministic feedback through tests, linters, and static analyzers; and workflow-orchestration tools coordinate multiple tools, roles, and lifecycle policies into a reliable execution process. Lifecycle hooks enforce permission checks, output sanitization, and memory updates before and after tool execution.

The Plan–Execute–Verify (PEV) loop reframes feedback-guided debugging as a broader control process. The harness first externalizes an intended change and its validation criteria as a contract, then executes the change inside a sandboxed and permissioned environment, and finally verifies the resulting state through deterministic sensors such as compilers, tests, static analyzers, and human-review gates. This loop unifies planning, execution, debugging, verification, and escalation into a single harness-level control process, where verification evidence determines whether to continue, revise, escalate, or roll back.

Agentic harness engineering treats the operating environment itself as an object of optimization. Deep telemetry records structured traces of model decisions, harness actions, environment states, and outcomes. An evolution agent uses this telemetry to diagnose failure modes, propose revisions to harness components (prompts, tool schemas, memory policies, permission rules, etc.), evaluate them on held-out tasks, and promote only changes that improve reliability without regressions. Governed harness mutation ensures that changes to safety-critical boundaries require human approval, subjecting the evolution agent itself to the PEV loop.

To scale to repository-level and multi-step tasks, the framework extends to multi-agent orchestration over code. Responsibilities are distributed across specialized roles—program synthesis, understanding, verification, execution, and planning—that interact through shared code artifacts and execution feedback. Interaction modes include collaborative synthesis, critique and repair, adversarial validation, and reasoning debate. Workflow topologies range from pre-defined chains and cyclic patterns to adaptive topologies that are dynamically generated or optimized per task. Execution feedback, from compiler errors to fine-grained simulation traces, provides objective signals that ground coordination. Shared-harness synchronization mechanisms, such as sequential handoff, blackboards, parallel branches with merge, and structured context scheduling, maintain a consistent view of program state across agents. The authors argue for a shared code-centric harness substrate—a formal, persistent representation of the program environment that agents can query and update—as the foundation for robust, scalable multi-agent intelligence.

Experiment

The surveyed works establish a shift from static text-based benchmarks to code-grounded evaluation environments where agents interact with executable systems, receiving runtime feedback and being assessed through verifiable state transitions rather than output matching. These setups validate that code serves as a unifying substrate for perception, action, and evaluation, enabling scalable, reproducible testing across software engineering, GUI control, and embodied tasks. However, end-task success metrics conflate model and harness capabilities, motivating the need for harness-level evaluation that measures execution reliability, feedback quality, and oracle adequacy to ensure that evaluators capture intended tasks rather than narrow proxies.

Code serves as a reasoning substrate through two main paradigms: delegated systems that generate code to augment language model reasoning, and formal systems that interact with symbolic solvers or proof assistants for machine-checkable outputs. As code becomes the unifying medium for agent perception, action, and world models, evaluation must shift from end-task accuracy to harness-level metrics that assess the runtime system's reliability, feedback quality, and safety. Delegated approaches like PoT and PAL merge code with natural language reasoning to decouple logic from computation, while formal methods such as SATLM and ReProver use external solvers and proof assistants to provide verifiable reasoning backends. Current evaluations that only measure final task success conflate model capability with harness quality, motivating new metrics for trajectory efficiency, verification strength, recovery ability, state consistency, safety compliance, and replayability.

Code is increasingly used as an action interface, with systems like AutoHarness synthesizing harnesses to validate actions and others grounding language model plans in physical feasibility, uncertainty calibration, or lifelong skill reuse. This shift toward end-to-end agents and executable world models makes code the common medium for observations, actions, and evaluation, but evaluating such code-agent harnesses requires new metrics beyond end-task success to capture harness quality, feedback reliability, and oracle adequacy. AutoHarness generates code harnesses that filter invalid environment interactions, focusing on action validation. SayCan and KnowNo select skills based on physical feasibility and calibrated planner uncertainty, respectively. SkillVLA and BOSS extend grounding to combinatorial skill reuse and synthesize new skill chains through guided practice. LRLL enables lifelong grounding via memory and self-exploration, evolving the skill interface over time. End-task success metrics conflate base model capability with harness quality, prompting a need for harness-level dimensions like trajectory efficiency and verification strength. Oracle adequacy is a central bottleneck because narrow executable proxies may not capture the full intended task.

Systems that represent environments as code divide into structured methods, which encode environments as explicit code artifacts like object hierarchies or renderable HTML, and trace-based methods, which leverage program execution traces for semantic alignment or model-based reinforcement learning. The innovations range from visual scene encoding to training large language models directly on execution traces. Structured approaches include ViStruct, which encodes visual scenes as data structures, and Code2World, which predicts GUI states as renderable HTML for reinforcement learning. Trace-based systems like WorldCoder synthesize transition and reward models from execution traces, while CWM trains large language models natively on program execution traces.

Code agent planning modules span linear decomposition, structure-grounded, search-based, and orchestration-based categories, with feedback ranging from none to combined critique and test signals. The shift toward end-to-end agents that internalize planning and grounding is displacing these modular pipelines, making it critical to evaluate the entire runtime harness rather than only end-task success. Planning modules range from linear decomposition with no feedback (e.g., Self-Planning) to search-based and orchestration-based methods that integrate execution environments and combine critique with test signals (e.g., ReThinkMCTS, MapCoder). End-to-end agents that internalize perception, planning, grounding, and action are replacing modular planning modules, shifting evaluation needs toward harness-level metrics like feedback quality and trajectory efficiency.

Code-agent harnesses use distinct memory mechanisms to manage state across long-horizon software engineering tasks. Working memory tracks repair trajectories and runtime state to ground edits, semantic memory retrieves repository structure and code evidence for context-aware generation, and experiential memory replays past trajectories to reuse quality experiences while filtering noise. These approaches externalize memory as a governable system component rather than relying solely on conversational history. Working memory methods (SWE-agent, CodeMem, RepairAgent) manage repair trajectories, context slots, and bug evidence through structured state tracking, budgeted slot management, and dynamic prompt-state updates. Semantic memory approaches (AutoCodeRover, RepoCoder, CodeRAG) ground localization and patching in repository structure via structure-aware retrieval, iterative retrieval, and multi-path reranking. MemGovern introduces experiential memory, replaying trajectories, reflections, and critiques with governed replay to reuse quality experience while filtering noise. Memory is increasingly treated as an externalized, retrievable, and traceable state-management layer rather than a simple accumulation of conversation history.

The experiments examine code as a reasoning substrate, action interface, environment representation, planning module, and memory layer, each validating the shift toward end-to-end agents where code unifies perception, action, and world models. They demonstrate that evaluating only end-task success conflates model capability with harness quality, motivating new metrics such as trajectory efficiency, verification strength, recovery ability, and oracle adequacy. Structured and trace-based environment encodings, planning modules with varying feedback, and externalized memory mechanisms all illustrate that the runtime harness's reliability, feedback quality, and safety are critical. Overall, the findings advocate for harness-level evaluation that captures the system's robustness and safety beyond final accuracy.