One-sentence Summary

Researchers from the University of Illinois Urbana–Champaign and Meta AI propose EvoHarness-RL, a method that learns self-evolving runtime harness policies for long-horizon LLM agents through supervised harness fine-tuning and cost-aware GRPO, coordinating selective access to Belief, Progress, and Experience (BPE) state, achieving 96.9% success on ALFWorld with Qwen3-8B and revealing harness annealing and evolution.

Key Contributions

• EvoHarness-RL is introduced as a trainable coordination layer that structures the external workspace as Belief, Progress, and Experience (BPE) and defines harness meta-actions for runtime state construction and updating.
• A two-stage training recipe uses supervised harness fine-tuning on expert demonstrations to bootstrap harness use, followed by cost-aware GRPO to optimize selective read, update, and consolidation policies for external state.
• On ALFWorld with a Qwen3-8B model, EvoHarness-RL reaches 96.9% success, and analysis reveals harness annealing (internalizing harness-use patterns into selective external-state access) and harness evolution (experience consolidation creating a compact, task-adaptive state substrate).

Introduction

LLM agents performing long-horizon tasks rely on an external harness of tools, memory, verifiers, and state trackers to maintain context, track progress, and recover from errors. While these components are increasingly sophisticated, the agent’s policy for accessing them is typically fixed by prompts or heuristics, leaving the agent unable to learn when to read, update, or consolidate external state as part of its own decision-making. The authors propose EvoHarness-RL, a trainable coordination layer that unifies the harness into a Belief, Progress, and Experience workspace and equips the agent with compact meta-actions to query and modify it. A two-stage training recipe first bootstraps harness use from demonstrations and then, via cost-aware reinforcement learning, optimizes the tradeoff between task success and the interaction budget consumed by harness actions, turning harness access from a static scaffold into a learned runtime policy.

Method

The authors introduce EvoHarness-RL, a trainable coordination layer that enables an agent to actively manage an external harness workspace during long-horizon tasks. The system comprises a unified BPE state abstraction, a compact agent-harness action protocol, an environment adapter that grounds the abstraction in a specific domain, and a two-stage cost-aware training pipeline. Together, these components allow the policy to decide when to read from or write to the harness, trading off information access against interaction cost.

The harness state is structured into three functional roles: Belief, Progress, and Experience (BPE). At each step $t$t, the harness renders

$$\mathcal{H}_t = (B_t, P_t, E_t),$$Ht​=(Bt​,Pt​,Et​),

where $B_t$Bt​ stores task-relevant facts inferred from interaction (e.g., object states and locations), $P_t$Pt​ maintains a list of subgoal-status records $(g_i, \sigma_i)$(gi​,σi​) that externalize what has been attempted and what remains open, and $E_t$Et​ holds cross-episode knowledge such as skills, failure modes, and search priors. This compact representation addresses recurrent failure modes in long-horizon execution: losing track of the current environment state, forgetting progress, and repeatedly rediscovering known procedures.

To interact with the BPE workspace, the policy uses a small set of meta-actions

$$\mathcal{A}_{\mathrm{bpe}} = \{\text{track}, \text{commit}, \text{recall}, \text{note}\},$$Abpe​={track,commit,recall,note},

which respectively read from $B_t$Bt​, update subgoal status in $P_t$Pt​, retrieve reusable knowledge from $E_t$Et​, and record new insights for later consolidation. The full action space at step $t$t is the union of environment actions and harness actions, $\mathcal{A} = \mathcal{A}_{\mathrm{env}} \cup \mathcal{A}_{\mathrm{bpe}}$A=Aenv​∪Abpe​. Given the observation $o_t$ot​, the rendered harness state $\mathcal{H}_t$Ht​, and task context $c_t$ct​, the policy samples

$$a_t \sim \pi_\theta(\cdot \mid o_t, \mathcal{H}_t, c_t).$$at​∼πθ​(⋅∣ot​,Ht​,ct​).

All actions consume the same interaction budget, so the agent must learn when querying the harness is worth its cost.

The BPE interface is functional rather than implementation-specific, so an environment adapter bridges domain signals to the general abstraction. The adapter processes observations, action results, and verifier feedback, maintains internal stores for belief, progress, and experience, and renders the policy-facing triplet $(B_t, P_t, E_t)$(Bt​,Pt​,Et​). It also grounds the four harness actions in the target domain. In the ALFWorld instantiation, Belief is a world-state store updated after each environment step; the policy issues track[object] or track[world] to inspect it on demand. Progress is a bounded list of subgoal-status entries, populated by commit[subgoal] to externalize execution steps. Experience is a cross-episode skill store organized into general skills, task-specific skills, common mistakes, and object-location search priors. The policy uses recall[query] to access prior knowledge and note[insight] to queue new lessons. During parallel rollout collection, the main skill store is fixed within a batch, and a consolidation model merges accumulated notes and trajectory summaries into the store at epoch boundaries.

The policy is trained in two stages. First, supervised fine-tuning (SFT) bootstraps the model on teacher trajectories that intermix environment actions and BPE harness actions. The teacher demonstrations teach the model both task-solving behavior and the basic semantics of when to track, commit, recall, or note, while the experience collected during teacher rollouts initializes the skill store. Second, the policy is optimized with Group Relative Policy Optimization (GRPO), using a cost-aware reward that combines a sparse success signal with auxiliary shaping terms:

$$R(\tau) = \underbrace{R_{\mathrm{succ}}(\tau)}_{\text{task success}} + \underbrace{\lambda_{\mathrm{eff}} R_{\mathrm{eff}}(\tau)}_{\text{efficiency bonus}} + \underbrace{\lambda_{\mathrm{div}}(u) R_{\mathrm{div}}(\tau)}_{\text{action diversity}} - \underbrace{\lambda_{\mathrm{spam}} R_{\mathrm{spam}}(\tau)}_{\text{spam penalty}} - \underbrace{\lambda_{\mathrm{inv}} R_{\mathrm{inv}}(\tau)}_{\text{format penalty}}.$$R(τ)=task successRsucc​(τ)​​+efficiency bonusλeff​Reff​(τ)​​+action diversityλdiv​(u)Rdiv​(τ)​​−spam penaltyλspam​Rspam​(τ)​​−format penaltyλinv​Rinv​(τ)​​.

Task completion provides a sparse reward of $10$10 only when the episode is solved. The efficiency bonus $R_{\mathrm{eff}}(\tau) = \max(0, 1 - |\tau| / T_{\max})$Reff​(τ)=max(0,1−∣τ∣/Tmax​) is granted exclusively on success, penalizing redundant harness queries. To prevent policy collapse into ignoring harness actions or falling into repetitive loops, a time-dependent vocabulary diversity bonus is applied:

$$R_{\mathrm{div}}(\tau) = \frac{|\{\mathrm{verb}(a_t): a_t \in \tau\}|}{|\tau|}, \quad \lambda_{\mathrm{div}}(u) = \frac{\lambda_{\mathrm{div}}^{\max}}{2} \left(1 + \cos \frac{\pi u}{U}\right),$$Rdiv​(τ)=∣τ∣∣{verb(at​):at​∈τ}∣​,λdiv​(u)=2λdivmax​​(1+cosUπu​),

where $u$u is the current RL epoch and $U$U is the annealing horizon. This curriculum encourages broad exploration of harness actions early in training and gradually decays to force specialization. Fixed penalties $R_{\mathrm{spam}}$Rspam​ and $R_{\mathrm{inv}}$Rinv​ suppress degenerate repetitions and malformed syntax.

Experiment

Experiments on ALFWorld evaluate EvoHarness-RL against frozen and trainable baselines, ablate its belief-progress-experience harness, and test generalization to unseen environments. The findings confirm that all three harness components work synergistically and are essential for long-horizon tasks, while RL optimization enables the policy to internalize routine scaffolding, reduce costly harness calls, and robustly generalize beyond training distributions. The cross-episode experience store evolves from rapid accumulation into a compact, task-adaptive memory, reinforcing that the harness becomes a learned coordination mechanism rather than a static memory.

EvoHarness-RL with Qwen3-8B achieves state-of-the-art performance on the ALFWorld seen split, substantially surpassing all baselines and matching top frontier models. The BPE framework universally improves success rates across model scales, with the largest gains on weaker frontier models. Two-stage training from prompt scaffolding to supervised fine-tuning to reinforcement learning progressively lifts performance, validating the pipeline. EvoHarness-RL on Qwen3-8B attains a 96.9% average success rate, matching Claude Opus 4.5 and decisively outperforming trainable agents like SkillOS and SkillRL. Applying the explicit harness boosts GPT-4.1 by over 22 absolute points and GPT-5 by over 25 points, demonstrating broad benefits for struggling frontier models. Even Claude Opus 4.5, already near the performance ceiling, gains +2.1 points from the harness, reaching 98.5%. The progression from prompt-time scaffolding (56.4%) to supervised fine-tuning (68.6%) to reinforcement learning (96.9%) validates the two-stage training pipeline. The BPE framework provides consistent improvements across all model sizes, confirming that externalizing belief, progress, and experience is critical for long-horizon task execution.

Ablating any single BPE component lowers overall success, demonstrating that belief, progress, and experience act synergistically. Removing belief tracking hits object localization tasks hardest, disabling progress degrades long-horizon multi-step tasks, and discarding reusable experience causes the largest average drop, especially for complex state-change tasks. Without belief, success on Clean and Cool drops sharply, while Heat is unaffected, showing that explicit object tracking matters most for localization and state verification. Disabling experience yields the lowest overall average success rate (48.6%) and heavily impacts complex state-change tasks like Heat. Ablating progress disproportionately harms long-horizon dependent subgoal tasks such as Pick2.

On ALFWorld unseen tasks, a prompt-time harness lifts ReAct's average success rate from 50.0% to 77.6%, but supervised fine-tuning to imitate harness calls reduces generalization to 69.4%. The RL-optimized policy, trained with cost-aware GRPO, reaches 86.6%, showing it learns to selectively use the harness for robust performance in novel environments. The prompt-time harness strongly improves over ReAct on most tasks, but SFT degrades performance on Cool and Heat, suggesting imitation overfits to seen harness-use patterns. The RL-optimized policy achieves near-perfect scores on Look (94.4%), Cool (95.2%), and Pick2 (88.2%), substantially outperforming all other variants.

Experiments on ALFWorld seen and unseen tasks evaluate EvoHarness-RL, which externalizes belief, progress, and experience through a BPE harness and is trained via a two-stage pipeline ending with cost-aware reinforcement learning. The harness universally improves success rates across model scales, and ablations reveal that each component is essential, with belief, progress, and experience synergistically addressing localization, long-horizon planning, and complex state changes. On unseen tasks, the RL-optimized policy learns to selectively invoke the harness, achieving robust generalization while imitation overfits, confirming that explicit state tracking and selective tool use are key for long-horizon execution.