One-sentence Summary

Researchers from The University of Hong Kong and collaborators propose HiSTEVE, a hierarchical VLA agent for open-world games that combines a capability-aware harness with structured memory, a VLM planner, and a policy-coupled completion detector, enabling a single vision-only VLA to perform native keyboard-and-mouse control through regime-specific RMSNorm branches and separate mouse-movement token embeddings, and achieving $85.0\%$85.0% on MineStudio Bench and $37.4\%$37.4% average success rate on OpenHA.

Key Contributions

• HiSTEVE is a hierarchical VLA agent with a capability-aware harness that combines structured memory, a VLM planner, and a policy-coupled task completion state detector for long-horizon execution and recovery.
• A single VLA provides vision-only native keyboard-and-mouse control through regime-specific RMSNorm branches, separate mouse-movement token embeddings, and a three-stage training strategy of world knowledge injection, imitation learning, and reinforcement learning.
• HiSTEVE achieves state-of-the-art atomic-task performance with 85.0% on MineStudio Bench and 37.4% average success on OpenHA, and completes long-horizon tasks such as crafting an iron pickaxe from scratch.

Introduction

Building virtual embodied agents that can play open-ended games like Minecraft under the same keyboard-and-mouse constraints as humans requires both long-horizon autonomy and reliable native-interface control. Prior work tends to focus on only one side: visuomotor or vision-language-action models can map pixels to actions but struggle with persistent task tracking and recovery, while language-model planners often depend on environment APIs or predefined skills that are not available in most games. The authors propose HiSTEVE, a hierarchical vision-language-action agent that combines a capability-aware planning harness with a single low-level policy for native control. The harness maintains structured task memory, verifies completions, and decides when to advance, recover, or replan, while regime-specific normalization branches and separate mouse-movement embeddings help the policy handle both non-GUI and GUI interaction without negative transfer.

Method

The authors propose a capability-aware agent harness that coordinates a VLM planner and a low-level VLA policy for long-horizon autonomy. The harness integrates capability-constrained planning, evidence-driven handoff and recovery, and structured memory. The VLA policy performs native visuomotor control, while a policy-coupled task completion state detector supports the harness by determining when a VLA invocation should end.

The planning process is reference-free: no predefined subtask sequence or expert route is provided. At each decision step, the harness maintains a structured planning state that summarizes verified progress, unresolved requirements, and previous execution outcomes. It restricts the VLM planner to a set of parameterized atomic subtask schemas that are known to be executable by the VLA. Each schema encodes applicability conditions, semantic parameters, observable progress and completion conditions, recognizable failure conditions, and an execution budget. Given the long-horizon objective, current and recent visual observations, the planning state, the admissible schema set, and retrieved contextual memory, the VLM planner selects and instantiates the next subtask as

The output is a semantic objective rather than a callable skill or scripted action sequence. The harness converts it into a fixed natural-language instruction that the VLA executes through native keyboard-and-mouse actions within a bounded invocation.

For control handoff, the system uses an evidence-driven protocol. During each bounded VLA invocation, a task-completion state detector estimates a completion probability for the current instruction. When this probability exceeds a confidence threshold, or when the execution budget is exhausted, control returns to the harness. The mechanism detects candidate completion rather than execution failure. The harness then verifies progress and completion conditions using current and recent visual observations and records the outcome as success, failure, or interrupted. After verification, it advances when completion conditions are satisfied, attempts local recovery when the subtask remains recoverable, and replans from the updated state when recovery is infeasible, the subtask becomes inapplicable, or the visual state changes substantially.

The harness also maintains compact structured memory around verified state transitions. Working memory corresponds to the current planning state and records the active objective, the current atomic subtask, verified progress, unresolved requirements, and recent failures. Episodic memory stores planning states before and after each VLA invocation together with the selected subtask, outcome, execution cost, and diagnostics. Procedural memory stores reusable dependencies among parameterized subtasks derived from verified successful executions. World memory stores landmarks and spatial relations inferred from visual observations. At each high-level step, working memory is used directly, and relevant entries from the other stores are retrieved to form the contextual memory.

The lower-level VLA policy maps a subtask instruction and first-person observations to native keyboard-and-mouse actions. The policy is initialized from Qwen2.5-VL-7B and runs autoregressively at 20 Hz. Because non-GUI and GUI interactions require different visuomotor skills, the authors introduce Regime-Specific RMSNorm, or RS-Norm, which maintains separate learnable scale vectors for non-GUI and GUI interactions in the pre-attention and pre-MLP RMSNorm modules of each decoder layer. For interaction type $c\in\{\mathrm{NG},\mathrm{GUI}\}$c∈{NG,GUI}, the normalization at decoder layer $\ell$ℓ and module $q\in\{\mathrm{attn},\mathrm{mlp}\}$q∈{attn,mlp} is

where $\boldsymbol{\gamma}_{\ell,q}^{(c)}$γℓ,q(c)​ is the interaction-specific scale vector and all other parameters remain shared. This adds only $2Ld$2Ld parameters for a model with $L$L decoder layers and hidden size $d$d.

For action representation, the policy uses discrete action tokens in a VPT-style action space. Keyboard and click inputs are represented as discrete choices, while mouse displacements are transformed and quantized into categorical bins. To resolve ambiguity between camera rotation in non-GUI interactions and cursor movement in GUI interactions, the authors assign disjoint mouse-displacement token sets and insert an interaction-type token at the beginning of the sequence. These tokens decode to the same physical action codebook, so branching occurs only at the representation level. This design is referred to as Dual-Mouse.

The VLA is trained in three stages. The first stage injects world knowledge, equipping the policy with Minecraft concepts and visual grounding. The second stage uses imitation learning to teach instruction-conditioned native keyboard-and-mouse control. The third stage uses reinforcement learning with environment feedback to improve reliability, especially for precise GUI interactions. For a given initial condition, the trainer samples multiple trajectories, scores them using certified completion, maximum verified progress, execution cost, and failure penalties, and computes group-relative advantages:

The method optimizes informative branch groups with clipped GRPO-style updates and also constructs local preference pairs from successful and failed branches to target divergent action segments with a DPO-style loss. Non-GUI capabilities are preserved by interleaving world-action replay and regularizing against a fixed reference policy through

The task completion state detector is policy-coupled and shares the VLA image encoder. It uses a Qwen2.5-VL-3B backbone and conditions on VLA policy context by injecting the final-layer hidden states preceding the first action token through cross-attention after every fourth detector decoder layer. For detector hidden states $Z_t^{(\ell)}$Zt(ℓ)​ at layer $\ell$ℓ, the injection follows

The detector processes historical frames sampled before the query, prunes feed-forward network widths to roughly one third, and maps the final-token hidden state to two logits to produce a completion probability. The detector emits a candidate completion signal only when the probability exceeds a threshold for two consecutive queries.

Experiment

HiSTEVE is evaluated with GPT-4o as the planner and Qwen2.5-VL-7B as the low-level VLA on MineStudio Bench and OpenHA for atomic execution and on LongBench for long-horizon agency. The results show consistent gains over native-control and OpenHA baselines, with the largest improvements on embodied atomic tasks and on long-horizon resource, crafting, and exploration tasks, while multi-step GUI manipulation, recovery, and open-ended objectives remain difficult. A specialized completion detector provides more balanced and accurate task-completion signals than prompted general-purpose VLMs, and ablations indicate that regime-specific representations, staged policy training, and harness orchestration each contribute, with the harness being especially important for long-horizon coordination.

HiSTEVE delivers large atomic-task gains on MineStudio, with broad improvements over the OpenHA Text/CoA baseline and the largest absolute gain on Craft, which remains the weakest category. On OpenHA, the average gain is modest and concentrated in embodied tasks, while GUI and combat remain slightly below baseline and multi-step GUI execution persists as a bottleneck. LongBench final-goal success is very low among earlier listed methods, and the complete HiSTEVE harness achieves markedly higher long-horizon performance, driven more by orchestration and verification than by executor training alone. The biggest atomic improvement occurs on Craft, yet Craft remains the lowest-scoring MineStudio category, suggesting multi-step GUI manipulation is more difficult than mining or combat. Long-horizon success improves mainly from the agent-level harness rather than from stronger low-level execution, with earlier baselines showing very low final-goal success.

On a balanced atomic-subtask completion detection benchmark, a specialized 1.4B detector outperforms prompted general-purpose vision-language models. General-purpose models range from strongly favoring the incomplete class to moderate completion detection, with GPT-5.5 as the strongest baseline. The specialized detector achieves the highest accuracy and F1 while maintaining balanced precision and recall and the lowest false-positive and false-negative rates. The specialized detector achieves higher accuracy and F1 than all prompted VLM baselines while using fewer parameters than the Qwen3.5 models. Qwen3.5 baselines strongly favor incomplete predictions, resulting in very low recall and F1. Among general-purpose models, GPT-5.5 provides the strongest completion detection performance but still falls below the specialized detector. The specialized detector reduces both false-positive and false-negative rates relative to GPT-4o and GPT-5.5, indicating fewer premature and missed completions.

This ablation isolates staged policy training, regime-specific representation, and agent harness orchestration. Regime-specific representation and world knowledge injection yield clear gains on atomic benchmarks and long-horizon goal success, while environment-feedback RL mainly improves atomic execution. The complete harness provides only modest atomic changes but substantially improves long-horizon success and diagnostic progress. Shared representation is the weakest configuration, and adding regime-specific normalization plus dual-mouse handling produces complementary gains across MineStudio, OpenHA, and LongBench goal success. World knowledge injection improves lower-level execution on MineStudio and OpenHA and raises LongBench goal success, but further environment-feedback RL improves atomic performance without additional long-horizon goal gains. With the learned executor fixed, the full agent harness modestly changes atomic benchmark averages while markedly improving LongBench goal success and hidden diagnostic progress.

Across MineStudio, OpenHA, and LongBench, HiSTEVE achieves the largest atomic-task gains on MineStudio, especially on the difficult Craft category, while long-horizon success improves mainly through orchestration and verification rather than through low-level executor training. In a balanced atomic-subtask completion detection benchmark, a specialized 1.4B detector outperforms prompted general-purpose vision-language models, which often favor the incomplete class, yielding higher accuracy and F1 with lower false-positive and false-negative rates. Ablations show that regime-specific representation and world knowledge injection improve both atomic and long-horizon performance, environment-feedback RL mainly boosts atomic execution, and the full agent harness provides the largest gains in LongBench goal success and diagnostic progress.