One-sentence Summary

Salesforce AI Research propose StateAct, a code-first multi-agent harness that augments computer-use agents with direct program state access, delegates only necessary visual tasks to a GUI subagent, and verifies results via a finish gate, lifting Claude Opus 4.8 on OSWorld 2.0 from $20.6\%$20.6% to $26.9\%$26.9% binary success and $54.8\%$54.8% to $61.6\%$61.6% partial success at $\sim 9\times$∼9× lower cost than a screenshot-driven counterpart, while a code-only variant without the GUI subagent achieves just $45.9\%$45.9% partial, thus refocusing the bottleneck from perception toward reasoning.

Key Contributions

• The paper proposes StateAct, a code-first multi-agent harness that grounds the main agent in program state and delegates visual interaction to a dedicated GUI subagent for the few subgoals that require it, with only 28 of 108 tasks needing such interaction and just 1.1% of main-agent steps delegated.
• It introduces an independent finish gate that directly inspects the saved deliverable for structural errors such as missing or misdirected output, providing a narration-free, general verification mechanism that applies across all tasks without per-task engineering.
• On OSWorld 2.0, StateAct raises binary success from 20.6% to 26.9% and partial success from 54.8% to 61.6% with Claude Opus 4.8 at ∼ 9× lower cost, while ablation and diagnostics show that the improvement comes from observing state rather than added agent depth, shifting the primary limitation from perception to reasoning.

Introduction

Current computer-use agents operate real desktop software but predominantly rely on screen perception — reading and acting on pixels. While this works for short interactions, pixel-based observation becomes brittle on long-horizon tasks where subtle state differences (such as formula vs. literal cell values) are invisible, and small misreads compound across hundreds of steps. Moreover, screenshots provide no dependable signal that a task’s final artifact is correctly complete. The authors introduce StateAct, which reorients the agent loop around program state as the primary interface, using a dedicated GUI subagent only when programmatic access is impossible. This state-grounded design includes a verification gate that directly inspects the persisted artifact, and a context-management strategy to maintain task focus across long horizons.

Method

The authors establish a foundational principle based on two observation channels for a computer program state $s \in S$s∈S. The pixel channel is a render map $o_{\mathrm{pix}} = f_{\mathrm{render}}(s)$opix​=frender​(s), while the state channel is a query map $o_{\mathrm{state}} = g(s)$ostate​=g(s) (e.g., shell commands, DOM serialization). Rendering is lossy and non-injective, meaning $f_{\mathrm{render}}^{-1}$frender−1​ does not exist, and distinct states can produce indistinguishable screenshots. Over long horizons, this per-step lossy proxy accumulates state drift. Conversely, the state channel is effectively invertible over the substate a task touches. Furthermore, the deliverable of a desktop task is a change to program state, not a screenshot. For tasks depending on non-rendered content, the success predicate $G(s)$G(s) cannot be recovered from $f_{\mathrm{render}}(s)$frender​(s) alone.

As shown in the figure below, the pixel path relies on rendered, indirect observations and coordinate-based interactions, whereas the state path targets and queries the exact artifact directly.

To operationalize this principle, the authors design StateAct, which comprises three core components: a main agent acting through code on program state, an independent finish gate verifying completion by re-reading real artifacts, and context management sustaining the run over hundreds of steps.

Refer to the framework diagram for the overall system design.

The main agent's action space consists of code and structured operations, such as persistent bash, a file editor, and a plan checklist. No live screen actuation (e.g., mouse or keyboard) is exposed to the main agent. The agent performs state discovery by leveraging priors about how applications store state on disk and active probing (e.g., find, grep, sqlite3) to locate where state lives. When a subgoal is irreducibly visual (e.g., dragging a canvas element or dismissing a non-scriptable modal), the main agent delegates to a dedicated GUI subagent. Browser work is handled by a web subagent that navigates, executes JavaScript, and serializes the DOM to markdown. This delegation rule ensures the main agent remains state-grounded, with GUI usage accounting for only a small fraction of steps.

When the main agent calls finish, an independent finish gate spawns. Its context contains only the verbatim task instruction and machine access, with editor mutations blocked. It never sees the main agent's message history or plan. This design provides four key properties: it is narration-blind, state-grounded (checking the exact persisted artifact), anti-capture (rejecting evidence found only in side files), and allows bounded correction (up to three rounds of retries).

The figure below illustrates a condensed transcript of the finish gate catching a structural defect where the persisted store did not match the agent's claim.

To handle long episodes without overflowing context, the authors employ three mechanisms. First, fresh-context specialists run delegations in their own context windows, returning concise reports. Second, auto-compaction summarizes the oldest prefix near the context limit and strips images. Third, an externalized plan checklist persisted outside the message history is re-injected each turn to anchor multi-part tasks.

The figure below displays three successful trajectories illustrating state-first action, GUI delegation, and a gate retry.

Experiment

StateAct is evaluated on a long-horizon GUI benchmark using a strong language model backbone. Replacing the typical computer-use agent with state-grounded execution improves accuracy while cutting token cost by an order of magnitude, driven mainly by acting on state rather than screen, with a finish gate and context management providing additional benefits. Failure analysis shows that state-grounded code eliminates perception errors and the gate catches structural defects, but the dominant residual errors are reasoning failures that a prompt-level verifier cannot correct, revealing a value-verification ceiling. Ablations confirm flat delegation suffices, and a compact specialist can handle rare GUI interactions on shorter tasks, though the hardest long-horizon desktop tasks still benefit from a frontier GUI subagent.

On the OSWorld 2.0 benchmark, the StateAct state-grounding harness improves Claude Opus 4.8's binary success from 20.6% to 26.9% and mean partial success from 54.8% to 61.6%, surpassing all prior public systems. This gain is accompanied by a roughly ninefold reduction in per-task cost, making StateAct the top-performing and most cost-efficient entry. StateAct delivers a 6.3-point absolute improvement in binary success and a 6.8-point improvement in partial success over the same model using a pixel-based agent harness. With a mean partial success of 61.6% and a binary success of 26.9%, StateAct outperforms the previous best public system by over 12 partial points while costing only about $7.80 per task.

StateAct averages 155 model turns per task, substantially fewer than some other models like MiniMax M3 (327) but more than the base Opus 4.8 (103). Turn counts vary markedly by task capability, peaking for high-loop tasks and dropping for menu editing, reflecting the system's reliance on a subagent only when visual fallback is needed. StateAct (155 turns) uses more turns than Opus 4.8 (103) and GPT-5.5 (95) but far fewer than Sonnet-4.6 (253), GLM-5V (314), and MiniMax M3 (327). StateAct's turn count peaks at 282 for high-loop tasks and falls to 136 for menu-editing tasks, showing efficiency on structured interactions. The main agent contributes 57 turns while subagent delegation adds 98 interior turns, so most turns arise from GUI interactions not main-agent reasoning.

Ablation of components shows the full StateAct system reaches 26.9% binary and 61.6% partial success. The GUI subagent is the most critical component: removing it causes the largest drop in both metrics, with partial score falling by over 10 points. The finish gate and planning component also contribute, but the finish gate is limited to structural checks and misses most value errors. Removing the GUI subagent (—act) reduces partial score from 61.6 to 51.3 and binary from 26.9 to 18.5, the largest degradation among all ablations. Disabling the finish gate (—verify) lowers binary from 26.9 to 23.1; analysis reveals the gate correctly rejects only 8 of 76 reached tasks and wrongly passes 68, because it cannot catch value errors. Turning off the planning component (—sustain) yields a binary of 21.3, worse than the no-finish-gate setting, indicating sustained reasoning plays a substantial role beyond structural verification.

A bare bash-only agent without GUI, subagents, or finish gate achieves lower partial success than the reference, showing that code actions alone are insufficient. State-grounded actions (StateAct) improve binary and partial scores, and their advantage appears mainly on long-horizon tasks: on a short-horizon benchmark performance matches the reference, while on a weaker model the grounding provides a small lift. Bash-only configuration reaches 45.9% partial, below the reference 54.8%, indicating the scaffold is essential beyond raw code actions. On Claude Sonnet 4.6, state-grounding raises binary success from 8.3% to 11.1%, showing benefit even for a weaker backbone. On the short-horizon OSWorld-Verified, StateAct and the reference achieve similar binary scores (78.4% vs 77.3%), confirming grounding benefits concentrate on long-horizon tasks.

The finish gate correctly passes nearly all binary-success tasks but also passes the vast majority of non-perfect tasks because it checks only structural properties. It correctly rejects only a small fraction of non-perfect tasks that reach it, resulting in a high error rate driven by its inability to detect value or reasoning errors. Of 79 non-perfect tasks, the gate wrongly passed 68 and correctly rejected 8; 3 tasks never called finish and so were not evaluated by the gate. The gate's error rate on non-perfect tasks that reached it was approximately 90%, reflecting a structural-only verification ceiling. Among 29 binary-success tasks, the gate passed 28 and failed 1, showing high recall for structurally correct successful completions.

On the OSWorld 2.0 benchmark, the state-grounding harness StateAct raises Claude Opus 4.8’s binary and partial success substantially while cutting per-task cost by about ninefold, surpassing all prior public systems. Ablation experiments show the GUI subagent is the most critical component, whereas the finish gate only catches structural errors and misses value mistakes, and the planning module contributes meaningfully beyond verification. State-grounding’s advantage appears mainly on long-horizon tasks, and the system stays efficient by delegating GUI interactions to a subagent that accounts for most of the modest increase in model turns.