One-sentence Summary

Researchers from Zhejiang University, Nanjing University of Aeronautics and Astronautics, Cornell University, and others propose EmbodiedSkills, a unified framework that treats each skill decision as an execution proposal with prerequisite checks and outcome verification via a shared executable-skill interface, enabling replaceable low-level VLA policies and structured trajectory supervision; instantiated with Qwen3-VL and OpenPI/π, it achieves $86.20\%$86.20% average success across $50$50 RoboTwin 2.0 tasks and $97.40\%$97.40% across the $4$4 LIBERO suites.

Key Contributions

• Introduces EmbodiedSkills, a framework that treats each skill decision as an execution proposal, with a runtime checking prerequisites before execution and verifying outcomes afterward through a shared executable-skill interface that unifies high-level planning, bounded low-level VLA execution, and post-action verification in one agent loop.
• Records planning, execution, verification, and recovery events as structured trajectories, providing supervision for training individual components and enabling optional online adaptation, while allowing low-level VLA policies to be replaced or adapted without altering the agent loop.
• Achieves 86.20% average success across 50 RoboTwin 2.0 tasks, 97.40% across the four LIBERO suites, and 12.5% on four memory-dependent RMBench tasks with task-adapted low-level VLA policies, demonstrating that a common subgoal-conditioned execution interface supports strong action policies across distinct manipulation benchmarks while preserving explicit control structure.

Introduction

Vision language action (VLA) models map visual observations and language instructions directly to robot actions, and recent large-scale pretraining has made them increasingly versatile across instructions, embodiments, and long-horizon tasks. However, long-horizon manipulation demands more than next-action prediction: a robot must perceive the scene, select subgoals, check executability, invoke low-level policies, verify outcomes, and recover from failures. End-to-end VLA policies leave these intermediate decisions implicit, making failures hard to diagnose, while LLM-based agents make decisions explicit but cannot guarantee they are valid or executable in the current physical state, especially under partial observability and contact-rich failures. The authors propose EmbodiedSkills, a unified framework that structures perception, planning, execution, verification, and recovery around executable embodied skills, each defined by typed inputs, outputs, and prerequisites. A high-level agent policy proposes structured skill calls, a runtime enforces prerequisites and legal state transitions before execution, and a low-level VLA policy generates bounded action chunks, with post-execution verification signals feeding back into subsequent decisions. This policy-runtime separation makes failures explicit and traceable, and the shared skill interface allows independent training and replacement of planner, verifier, and VLA components. The authors instantiate EmbodiedSkills with Qwen3-VL-based components and OpenPI/pi_0.5, achieving 86.20% average success across 50 RoboTwin 2.0 tasks and 97.40% across LIBERO suites, demonstrating strong cross-benchmark execution while preserving explicit control structure.

Dataset

The authors construct training examples that mirror the exact input format used by the deployed system. Each sample is defined as a tuple: task instruction, current phase, visible images, compact runtime state, admissible skills, and recent loop history. The output label is the component's decision at that time step.

• Component-specific inputs: Each component (planner, verifier, scheduler) receives only the fields relevant to its interface. For example, the planner gets planning context, the verifier gets the active subgoal and post-execution evidence, and the scheduler gets admissible choices plus latest artifacts and errors.
• Context retention: Skill outputs, environment feedback, previous decisions, and runtime errors are kept as conditioning context in their original temporal order. The same history-compaction rule is applied during both data construction and deployment, ensuring no component is trained with evidence that would be absent at test time.
• Loss application: For vision-language supervision, loss is applied only to tokens generated by the learned component. Prompts, images, tool results, and environment messages are treated as context, not prediction targets, preserving the distinction between a policy proposal and runtime-supplied evidence.

Method

3 Methodology

EmbodiedSkills formulates an embodied VLM–VLA system as a guarded finite-stage controller over executable embodied skills. The high-level agent policy reads the task, current visual evidence, loop state, and phase-admissible skills, then selects one structured operation. The low-level VLA policy maps the active subgoal, current observation, and robot state to a bounded action chunk. The runtime checks every proposed operation, records its result as an explicit artifact, invalidates dependent artifacts when their context changes, and exposes the updated state to the next decision. Model, environment, and VLA policy adapters preserve this interface across concrete instantiations.

3.1 Agent State and Skill Decision

An episode starts from a natural-language instruction $x$x and an environment $E$E. At loop step $t$t, the method-level state is

$$s_{t} = (z_{t}, \mathcal{M}_{t}, \mathcal{H}_{t}),$$st​=(zt​,Mt​,Ht​),

where $z_{t}$zt​ is the current phase, $\mathcal{M}_{t}$Mt​ is the set of available task artifacts, and $\mathcal{H}_{t}$Ht​ is the ordered loop trace. Artifacts may include the current observation, optional perception and grounding results, world state, complete task plan, active subgoal, preflight evidence, action chunk, execution report, verification report, and recovery context.

The policy receives a deployment-consistent compact context

$$C_{t} = \Psi(x, z_{t}, \mathcal{M}_{t}, \mathcal{H}_{t}),$$Ct​=Ψ(x,zt​,Mt​,Ht​),

where $\Psi$Ψ retains the complete plan, active subgoal, current artifact summaries, recent errors, and an ordered, bounded summary of recent decisions and skill results. Older entries are compressed or dropped under a fixed history budget, while the current plan and subgoal remain explicit. Raw simulator internals and unbounded logs are not inserted into the policy context.

The policy chooses a structured decision from a state-dependent action set:

$$d_{t} \sim \pi_{\theta}(\cdot | C_{t}, z_{t}, \mathcal{A}_{t}), \qquad \mathcal{A}_{t} = \mathcal{G}_{\mathrm{state}}(\mathcal{K}_{z_{t}}, s_{t}),$$dt​∼πθ​(⋅∣Ct​,zt​,At​),At​=Gstate​(Kzt​​,st​),

where $\mathcal{K}_{z_{t}}$Kzt​​ is the configured skill set for phase $z_{t}$zt​, and $\mathcal{G}_{\mathrm{state}}$Gstate​ exposes only choices compatible with the current artifacts. The policy has three control types:

$$d_{t} \in \{\text{RUNSkill}(k, q), \text{ADVANCESTAGE}, \text{FINISHRUN}\},$$dt​∈{RUNSkill(k,q),ADVANCESTAGE,FINISHRUN},

where $k$k is an admissible skill and $q$q is its payload. Skill execution, forward progression, and termination therefore share one structured decision interface, while evidence-dependent rerouting remains governed by the runtime.

3.2 Executable Skill Contract

Each embodied skill is represented by the contract

$$k = (\mathcal{X}_{k}, \mathcal{Y}_{k}, \mathrm{pre}_{k}, \mathrm{exec}_{k}, \mathrm{post}_{k}, \mathrm{fail}_{k}),$$k=(Xk​,Yk​,prek​,execk​,postk​,failk​),

where $\mathcal{X}_{k}$Xk​ and $\mathcal{Y}_{k}$Yk​ are typed input and output schemas, $\mathrm{pre}_{k}$prek​ defines prerequisites, $\mathrm{exec}_{k}$execk​ is the executable operation, $\mathrm{post}_{k}$postk​ specifies the resulting state update, and $\mathrm{fail}_{k}$failk​ maps failures to explicit status and evidence. Artifacts carry provenance and freshness information so that observations, plans, and actions are not silently reused after their dependencies change. The contract applies to both model-backed skills and deterministic operations and prevents a textual proposal from being mistaken for a physical state transition.

At invocation time, the contract defines which state and evidence a skill may consume; after invocation, it determines how outputs, side effects, and failures become part of the shared task state. A model-generated result is therefore treated as a proposal until its schema and prerequisites have been validated. Successful outputs become typed artifacts that can support later skills, whereas failures remain explicit evidence available to the next policy decision. This makes learned perception, planning, verification, and action generation composable without assuming that they have identical internal representations.

The same contract also defines the boundary of invalidation. When an observation, active subgoal, or execution result changes, only artifacts that depend on the changed evidence need to be refreshed. As a result, the loop can reuse still-valid context while preventing stale predictions from authorizing new physical actions. The skill contract thus serves simultaneously as a composition interface, a runtime validity boundary, and a structured source of training traces.

3.3 Phase-Structured Skill Space

The runtime uses the ordered phase set

$$\mathcal{Z} = (\text{OBSERVE, PLAN, PREFLIGHT, EXECUTE, VERIFY, RECOVER}).$$Z=(OBSERVE, PLAN, PREFLIGHT, EXECUTE, VERIFY, RECOVER).

These phases describe the semantic structure of the loop rather than a rigid one-pass program. The policy may revisit earlier phases when new observations, execution outcomes, or verification evidence invalidate the current plan.

Grounding is task- and policy-conditioned rather than globally fixed to a source–target pair. A task may require no explicit object binding, one interaction object, multiple objects, or multiple destinations. Direct language-conditioned VLA policies may operate from images, robot state, and subgoal text, whereas geometry-based controllers may additionally require explicit object or region bindings.

3.4 Guarded Runtime Transition

The policy proposes decisions, whereas the runtime defines their execution semantics. Because the three controls have different invariants, the paper use separate guards. For a skill proposal,

$$G_{\mathrm{skill}}(s_{t}, d_{t}) = \mathbf{1}[d_{t} = \text{RUNSKILL}(k, q)] \, \mathbf{1}[\mathrm{stage}(d_{t}) = z_{t}] \, \mathbf{1}[k \in \mathcal{K}_{z_{t}}] \cdot \prod_{r \in \mathcal{R}(k, q)} \mathbf{1}[r(s_{t}) = 1],$$Gskill​(st​,dt​)=1[dt​=RUNSKILL(k,q)]1[stage(dt​)=zt​]1[k∈Kzt​​]⋅r∈R(k,q)∏​1[r(st​)=1],

where $\mathcal{R}(k, q)$R(k,q) contains the prerequisites of the selected skill and payload. Forward progression is guarded by

$$G_{\text{advance}}(s_{t}, d_{t}) = \mathbf{1}[d_{t} = \text{ADVANCESTAGE}] \, \mathbf{1}[R_{z_{t}}(s_{t}) = 1],$$Gadvance​(st​,dt​)=1[dt​=ADVANCESTAGE]1[Rzt​​(st​)=1],

where $R_{z_{t}}$Rzt​​ summarizes whether the current phase has produced the evidence needed by its successor. Termination uses the separate control gate

$$G_{\mathrm{finish}}(d_{t}) = \mathbf{1}[d_{t} = \text{FINISHRUN}].$$Gfinish​(dt​)=1[dt​=FINISHRUN].

The guarded update is

$$s_{t+1} = \begin{cases} \mathcal{U}(s_{t}, k(q)), & G_{\text{skill}}(s_{t}, d_{t}) = 1, \\ \mathcal{N}(s_{t}), & G_{\text{advance}}(s_{t}, d_{t}) = 1, \\ \mathcal{F}(s_{t}), & G_{\text{finish}}(d_{t}) = 1, \\ \mathcal{B}(s_{t}, d_{t}), & \text{otherwise}, \end{cases}$$st+1​=⎩⎨⎧​U(st​,k(q)),N(st​),F(st​),B(st​,dt​),​Gskill​(st​,dt​)=1,Gadvance​(st​,dt​)=1,Gfinish​(dt​)=1,otherwise,​

where $\mathcal{U}$U records the operation result and invalidates stale dependents, $\mathcal{N}$N advances the loop, $\mathcal{F}$F terminates it, and $\mathcal{B}$B records a blocked decision and its evidence. This separation allows a learned policy to choose among meaningful operations without making the prompt itself responsible for execution safety or state consistency.

3.5 Bounded Action Execution

In Execute, the low-level VLA policy maps the active subgoal, current observation, robot state, and readiness evidence to an action chunk and an execution report:

$$(g_{t}, O_{t}, F_{t}) \longrightarrow a_{t} \longrightarrow e_{t},$$(gt​,Ot​,Ft​)⟶at​⟶et​,

where $g_{t}$gt​ is the active subgoal, $O_{t}$Ot​ is the current observation, $F_{t}$Ft​ is the preflight evidence, $a_{t}$at​ is the proposed action chunk, and $e_{t}$et​ records its observed outcome.

An action chunk is a bounded command sequence

$$a_{t} = (\tau_{t}, U_{t}, H_{t}, \eta_{t}), \qquad U_{t} = [u_{t}^{1}, \dots, u_{t}^{H_{t}}],$$at​=(τt​,Ut​,Ht​,ηt​),Ut​=[ut1​,…,utHt​​],

where $\tau_{t}$τt​ is the action type, $H_{t}$Ht​ is the horizon, and $\eta_{t}$ηt​ ties the chunk to its subgoal and observation context. Let $\Sigma_{p}$Σp​ denote the action schema exposed by policy module $p$p. The runtime accepts a chunk only when

$$R_{\mathrm{act}}(a_{t}, s_{t}) = \mathbf{1}[0 < H_{t} \leq H_{\max}] \, \mathbf{1}[a_{t} \models \Sigma_{p}] \cdot \mathbf{1}[\text{fresh}(a_{t}; g_{t}, O_{t})] \, \mathbf{1}[\text{valid}(U_{t})].$$Ract​(at​,st​)=1[0<Ht​≤Hmax​]1[at​⊨Σp​]⋅1[fresh(at​;gt​,Ot​)]1[valid(Ut​)].

This checks the policy-specific action type and dimensions, numerical validity, the execution horizon, and consistency with the current subgoal and observation. After a chunk is executed, the loop obtains fresh evidence before deciding what to do next. A semantic subgoal may therefore require multiple bounded chunks rather than being forced into a single action horizon.

3.6 Verification and Recovery

Verify evaluates only the active subgoal using evidence captured after the latest action chunk. It predicts a semantic route from

$$\mathcal{V} = \{\text{ADVANCE, CONTINUE, REOBSERVE, REPLAN, RECOVER, FINISH}\}.$$V={ADVANCE, CONTINUE, REOBSERVE, REPLAN, RECOVER, FINISH}.

The route either advances the plan, preserves the current subgoal for another bounded attempt, refreshes scene evidence, revises the plan, or enters recovery. Recovery uses the same explicit task state and failure evidence to produce a revised execution context, which is checked again before another physical action. Overall task success remains defined by the environment's evaluation protocol rather than by the local subgoal judgement.

3.7 Algorithmic Summary

Algorithm 1 summarizes the AgentLoop. At each step, the policy receives compact state and admissible skills. Runtime guards block decisions that lack valid evidence, and every operation result is appended to the ordered trace used by subsequent decisions.

3.8 Modular Interfaces

EmbodiedSkills combines a typed task state, phase-structured skill interfaces, runtime validation, trajectory logging, environment adapters, and action policy modules. Perception, planning, action generation, verification, and recovery may use a shared base model with stage-specific adapters or separate models, provided that their inputs and outputs obey the same skill and state contracts.

4 Training the Agentic Layer

EmbodiedSkills exposes planning, skill selection, execution, verification, and recovery through explicit interfaces. This structure allows the learned components to be adapted independently instead of requiring one end-to-end optimization procedure. In the instantiation, component-level supervision is the primary training mechanism. The same trajectory interface also supports optional closed-loop policy optimization when interaction data and a reliable environment evaluator are available.

4.1 Component-Level Supervised Adaptation

The planner is trained to map the task instruction and current visual context to an ordered sequence of executable semantic subgoals. A subgoal specifies what physical state should be reached, while the low-level VLA policy generates the action sequence used to reach it under runtime checks. The supervision therefore avoids embedding simulator-specific control details in the high-level plan.

The low-level VLA policy is adapted separately with subtask-level demonstrations. Each training example pairs the observation and robot state with the active subgoal and the corresponding action sequence. At deployment time, the policy receives the same type of subgoal-conditioned context through the Execute interface and emits a bounded action chunk. A semantic subgoal may require more than one chunk: after each chunk, fresh evidence is collected and the verifier decides whether execution should continue or the loop should advance.

For high-level control, the authors supervise a Qwen3-VL scheduler on deployment-consistent decision traces while keeping the low-level VLA policy frozen. Each example conditions on the task, visible observations, compact task state, recent ordered history, and the skills admissible in the current phase. The target is the phase-appropriate skill decision and its structured arguments, including whether the loop should continue the current subgoal, advance the plan, or request a revised context. Freezing the low-level VLA policy isolates this SFT stage from continuous-control learning: it improves state-conditioned scheduling and skill use without changing how low-level actions are generated.

Other learned decisions can use the same decomposition. For example, a verifier can be adapted from post-execution observations and subgoal-completion labels, while the outer scheduler can be adapted from valid skill choices and their runtime outcomes. These components may share a base vision–language model while using stage-specific adapters; the runtime contract remains fixed across adapters. EmbodiedSkills does not require all components to be trained jointly, and a deterministic or externally provided component can be used where appropriate.

4.2 Deployment-Consistent Training Samples

Training examples follow the same input boundary used by the deployed component. For a model decision at time $t$t, the authors write

$$x_{t} = (g, z_{t}, \mathcal{I}_{t}, \tilde{s}_{t}, \mathcal{K}_{t}, h_{t}), \qquad y_{t} = \text{the component decision},$$xt​=(g,zt​,It​,s~t​,Kt​,ht​),yt​=the component decision,

where $g$g is the task instruction, $z_{t}$zt​ is the current phase, $\mathcal{I}_{t}$It​ contains the images visible to that call, $\tilde{s}_{t}$s~t​ is the compact runtime state, $\mathcal{K}_{t}$Kt​ is the set of admissible skills, and $h_{t}$ht​ is the compact recent loop history. Components receive only the fields relevant to their interface. In particular, the planner receives planning context, the verifier receives the active subgoal and fresh post-execution evidence, and the scheduler receives admissible choices together with the latest artifacts and errors.

Skill outputs, environment feedback, previous decisions, and runtime errors are retained as conditioning context in their original temporal order. The same history-compaction rule is applied during data construction and deployment, so a component is not trained with evidence that would be absent at test time. For vision–language supervision, loss is applied only to the tokens generated by the learned component; prompts, images, tool results, and environment messages are context rather than prediction targets. This preserves the distinction between a policy proposal and evidence supplied by the runtime.

4.3 Optional Closed-Loop Adaptation

The full AgentLoop can additionally collect interactive trajectories. One episode contains model decisions interleaved with executed skills and observed outcomes:

$$\tau = ((x_{1}, y_{1}), \dots, (x_{T}, y_{T}), e_{1:T}, R),$$τ=((x1​,y1​),…,(xT​,yT​),e1:T​,R),

where $e_{1:T}$e1:T​ is the typed runtime trace and $R$R is supplied by the environment adapter together with explicit penalties for invalid agent decisions when applicable. Infrastructure failures are recorded separately from agent-policy failures. During agent-level adaptation, the low-level VLA policy can be held fixed so that optimization changes high-level decisions without changing the continuous controller at the same time.

Group-relative policy optimization is one supported mechanism. Given $M > 1$M>1 rollouts of the same task condition, the episode score can be normalized within the group as

$$A_{i} = \frac{R_{i} - \mu_{\mathcal{G}}}{\sigma_{\mathcal{G}} + \epsilon_{\mathrm{norm}}}.$$Ai​=σG​+ϵnorm​Ri​−μG​​.

For generated token $j$j of decision $t$t, let $\rho_{i,t,j}$ρi,t,j​ be the likelihood ratio between the updated policy and the rollout policy. A masked clipped objective is

$$\mathcal{L}_{\text{online}} = -\frac{1}{N_{\text{gen}}} \sum_{i,t,j} m_{i,t,j} \min(\rho_{i,t,j} A_{i}, \operatorname{clip}(\rho_{i,t,j}, 1 - \epsilon_{\operatorname{clip}}, 1 + \epsilon_{\operatorname{clip}}) A_{i}),$$Lonline​=−Ngen​1​i,t,j∑​mi,t,j​min(ρi,t,j​Ai​,clip(ρi,t,j​,1−ϵclip​,1+ϵclip​)Ai​),

where $m_{i,t,j}$mi,t,j​ selects generated policy tokens and $N_{\text{gen}} = \sum_{i,t,j} m_{i,t,j}$Ngen​=∑i,t,j​mi,t,j​. Episode-level attribution is necessarily coarse: an identical return is assigned to multiple decisions whose causal contributions may differ. The authors therefore treat online optimization as an optional refinement mechanism rather than as a substitute for component-level supervision or as the source of the headline results in Section 5.

Experiment

The task-adapted low-level VLA policies exposed through the EmbodiedSkills execution interface were evaluated on RoboTwin 2.0 (50 manipulation tasks) and four LIBERO suites (spatial, object, goal, long-horizon), with additional tests on RMBench memory-dependent tasks. On RoboTwin 2.0, the policies achieved a macro-average success of 86.20%, improving over the LingBot-VA reference by 3.46 points, with gains concentrated on contact-sensitive and multi-stage tasks. On LIBERO, the average success rose to 97.40% versus 96.85% for OpenPI, with the clearest advantage on the long-horizon suite. Controlled ablations showed that combining semantic subtask conditioning, intermediate verification, and adaptive continuation is essential, as removing verification or subtask awareness caused substantial performance drops, while a one-chunk-per-subtask allocation proved insufficient.

EmbodiedSkills structures robot task execution into six phases, each with specific operations and exit evidence, enabling adaptive and verifiable progress. The execution interface is benchmark-agnostic and supports memory-dependent tasks, while ablations show that combining semantic subtasks with post-action verification and adaptive continuation is essential for performance. Each phase requires valid evidence before moving on, ensuring that actions are grounded in current observations and task context. The execution interface works across different benchmarks and task types, including tasks that depend on interaction history. Removing intermediate verification reduces performance significantly, showing that checking progress after each action is critical. Using the original whole-task instruction instead of active subtask conditioning further degrades performance, indicating the need for explicit stage information. Allowing multiple bounded action chunks per subtask is necessary, as many valid subtasks cannot be completed in a single chunk.

The task-adapted subtask-level policy outperforms all policy baselines and generalist VLA references on RoboTwin 2.0, achieving a macro-average success of 86.20% compared to 82.74% for the best reference. Gains are concentrated on contact-sensitive and multi-stage tasks, with improvements on 39 of 50 tasks and only minor regressions on tasks already above 90% success. The task-adapted policy improves over the strongest VLA baseline by 3.46 percentage points on average. Largest gains are seen on tasks like Hanging Mug, Blocks Ranking Size, and Open Microwave, with improvements up to 20 percentage points. Regressions are mostly limited to 1-2 percentage points on tasks where the baseline success is already above 90%.

The proposed VLA policy instantiation outperforms the official OpenPI reference on all four LIBERO suites, with an average success rate of 97.40% versus 96.85%. The largest improvement is on LIBERO-Long, while gains on Spatial, Object, and Goal are small due to near-saturation in the baseline. Average success rate improves by 0.55 percentage points over the official OpenPI reference. The clearest gain is on LIBERO-Long, rising from 92.4% to 93.6%. Improvements are consistent across all suites, with smaller margins on Spatial, Object, and Goal due to already high baseline performance.

On RMBench memory-dependent tasks, the proposed task-adapted policy achieves the highest macro-average success rate compared to published baselines, with notable gains on Press Button and Cover Blocks while remaining competitive on Battery Try and Blocks Ranking. The results indicate that the execution interface generalizes to tasks where actions depend on interaction history. The task-adapted policy outperforms all published baselines on the macro average, with the largest relative gains on Press Button and Cover Blocks. On Battery Try, the proposed policy matches the best published result, while on Blocks Ranking it is close to the top baseline. Published baselines often fail completely on Cover Blocks and Press Button, whereas the proposed policy achieves non-zero success on both.

Ablation results show that the full AgentLoop configuration, which combines semantic subtask conditioning, intermediate verification, and adaptive chunking, achieves the highest success rates across tasks. Removing intermediate verification or semantic subtask conditioning substantially degrades performance, and limiting each subtask to a single action chunk leads to near-zero success on many tasks. Full configuration outperforms all ablated variants on most tasks, with the largest gaps on tasks like Blocks Ranking RGB and Place Mouse Pad. Removing intermediate verification reduces success by a large margin on many tasks, though some tasks like Click Alarmclock remain unaffected. Removing semantic subtask conditioning causes severe drops, often halving or quartering success rates compared to the full model. Allowing only one action chunk per subtask yields very low success rates, often below 10%, indicating that many subtasks require adaptive continuation.

The experiments validate the EmbodiedSkills execution interface across multiple benchmarks, showing that its task-adapted subtask-level policy consistently outperforms generalist VLA baselines and published references, with the largest gains on contact-sensitive, multi-stage, and memory-dependent tasks. Ablations confirm that combining semantic subtask conditioning, intermediate verification after each action, and adaptive multi-chunk continuation is essential, as removing any of these components leads to substantial performance drops, often near-zero success on complex tasks.