One-sentence Summary

Researchers from Show Lab, National University of Singapore, propose Show-Harness, an embodied harness that enables vision-language models to control robots through discrete semantic action units and embodiment-specific interpreters, thereby unlocking zero-shot control for closed-source VLMs and low-cost fine-tuning of open-source ones, and introduces GUMI, a GUI-based demonstration interface, to achieve robust generalization across tasks and embodiments.

Key Contributions

• Show-Harness is an embodied harness with a compact semantic action interface that lets vision–language models directly control robots by reasoning over discrete action units, which are deterministically grounded into physical motions by embodiment-specific interpreters.
• GUMI (GUI Manipulation Interface) extends the same semantic action space to GUI-based demonstration collection, enabling humans and agents to operate robots across embodiments without specialized teleoperation hardware.
• Extensive experiments show that Show-Harness-equipped VLM agents generalize robustly across tasks, embodiments, and environments, outperforming representative agentic and vision–language-action paradigms, and that the interface unlocks zero-shot control with closed-source frontier VLMs and efficient fine-tuning of small open-source VLMs.

Introduction

Foundation vision–language models (VLMs) already capture object recognition, spatial reasoning, and task decomposition, but this knowledge does not easily translate into physical robot behavior. Existing approaches either fine-tune VLMs into vision–language–action (VLA) models that regress embodiment-specific continuous actions, collapsing broad semantic knowledge into opaque sensorimotor mappings that require repeated adaptation, or they keep the VLM at a high level and delegate physical execution to hand-designed controllers, weakening the direct link between semantic intent and fine-grained motion. The authors introduce Show-Harness, a model-agnostic embodied harness that instead exposes a discrete set of semantic action units (directional moves and gripper commands) that VLMs can natively reason over while an interpreter deterministically grounds each unit into bounded robot motion. This interface enables closed-source frontier VLMs to perform zero-shot manipulation and allows lightweight VLMs to be efficiently fine-tuned, outperforming representative agentic and VLA paradigms across tasks, embodiments, and environments. The same action space also powers GUMI, a GUI-based teleoperation interface that lets humans and agents collect demonstrations without specialized hardware.

Method

The authors introduce Show-Harness, an embodied framework designed to translate the intelligence of foundation vision-language models into physically grounded robot behavior. The system operates within an iterative perception-reasoning-action loop, seamlessly connecting high-level semantic reasoning with low-level physical control. As shown in the figure below:

At each interaction step, given a language instruction $\ell$ℓ, the system captures an observation $o_t = (\mathcal{I}_t, p_t)$ot​=(It​,pt​), where $\mathcal{I}_t$It​ denotes the multi-view visual observation and $p_t$pt​ represents the proprioceptive state of the robot. The perception stage processes these raw sensor streams into a context suitable for model reasoning. Multi-view guidance directs the model on how to prioritize different camera perspectives, such as using a global view for scene-level context and a wrist-mounted view for fine-grained manipulation. Simultaneously, proprioception translates the internal robot state into concise textual feedback, including gripper height, displacement, and contact states.

The reasoning stage structures the task into actionable intermediate decisions. The authors employ a configurable set of reasoning plugins to refine the context. Subtask planning decomposes the instruction into an ordered sequence of subtasks with completion criteria, allowing the model to autonomously advance the plan. Situated planning defers uncertain decisions, selectively triggering replanning only when required information becomes observable. Additional plugins include action chunking to reduce model-query frequency during open-loop execution, adaptive stepping to balance efficiency and precision, and visual prompting to convert ambiguous verbal targets into visual references. These plugins collectively produce a reasoning-refined context $c_t$ct​:

$$c_t = \Phi_{\mathcal{P}}(\ell, o_t, h_t)$$ct​=ΦP​(ℓ,ot​,ht​)

where $h_t$ht​ is the compact interaction history.

The central model then selects a semantic action $a_t$at​ conditioned on this refined context:

$$a_t = \pi(c_t) \in \mathcal{A}$$at​=π(ct​)∈A

Here, $\mathcal{A}$A is a compact set of fine-grained, actionable units that defines the semantic interface between the model and the robot. This vocabulary includes incremental translation and rotation units, gripper commands, and a completion token. The design ensures that actions are interpretable, embodiment-agnostic, and visually grounded.

To execute these semantic decisions, an embodiment-specific interpreter $g_E$gE​ deterministically grounds the unit $a_t$at​ into executable robot control $u_t$ut​:

$$u_t = g_E(a_t; s_t)$$ut​=gE​(at​;st​)

The interpreter updates the 6-DoF Cartesian pose setpoint $s_t = (\mathbf{x}_t, Q_t)$st​=(xt​,Qt​) using calibrated increments for translation and rotation, mapping semantic directions into the motion frame of the specific embodiment. This isolation of low-level control within the interpreter allows the semantic model interface to remain unchanged across different robots.

The action stage augments execution with lightweight interaction memory and recovery mechanisms. An action history plugin carries recent actions into the next step to provide temporal memory and prevent oscillation. A failure recovery plugin automatically detects grasp failures, resetting the gripper and rolling back to the relevant subtask.

The shared semantic interface supports two complementary modes of robot control. In the zero-shot mode, a frontier model directly controls the robot without fine-tuning. In the fine-tuning mode, the authors adapt a small open-source model to predict semantic action units directly. Given demonstrations $\mathcal{D}$D, the policy minimizes the token-level cross-entropy of the target unit:

$$\mathcal{L}(\theta) = - \sum_{(\ell, o, h, a) \in \mathcal{D}} \log \pi_\theta \big(a \mid \Phi_{\mathcal{P}_{\min}}(\ell, o, h)\big)$$L(θ)=−(ℓ,o,h,a)∈D∑​logπθ​(a∣ΦPmin​​(ℓ,o,h))

where $\mathcal{P}_{\min}$Pmin​ retains only a minimal decision context. Because semantic actions are predicted through the native vocabulary without dedicated action heads, this approach enables lightweight adaptation and strong generalization.

To facilitate data collection for the fine-tuning mode, the authors develop GUMI, a graphical user interface-based manipulation interface. Since the semantic action space is discrete and directly operable, humans and agents can operate the robot using the same semantic units. At each step, the interface records the pre-execution observation and the selected semantic action, yielding policy-ready pairs. Because each semantic unit is deterministically grounded by the interpreter, the rollout retains corresponding low-level commands, allowing a single demonstration to train both semantic-action and continuous-control policies across different embodiments.

Experiment

The evaluation spans two robot platforms and ten diverse manipulation tasks, comparing zero-shot frontier VLMs and fine-tuned small VLMs against VLA, VLA-centric, and code-as-policy baselines. Show-Harness agents consistently generalize better across tasks, environments, and embodiments, and the semantic action interface enables physical adaptability (e.g., fine-grained control, multi-arm coordination) without retraining, as well as semantic adaptability like reasoning and in-context learning from demonstrations. Ablations confirm that explicit action conventions, multi-view guidance, and selective plugins are critical for robust performance, while the framework scales effectively from large zero-shot models to compact fine-tuned ones.

The Embodied Harness organizes a VLM-based robot control loop into perception, reasoning, and action plugins that translate sensor streams, manage plans, and execute actions. Key plugins such as Situated Planning, Action History, and Failure Recovery yield substantial gains by deferring decisions until sufficient evidence is available, preventing repetitive action loops, and detecting empty grasps. Action-space conventions provide most of the grounding, with semantic names serving as a useful prior, while the same interface generalizes across diverse manipulation tasks without redesign. Situated Planning defers uncertain decisions and boosts hidden-object search success from 35% to 85% by enabling targeted exploration instead of premature planning. Removing action history reduces success and increases timeouts due to oscillation between opposing actions; recording recent actions exposes loops and provides lightweight memory for stable closed-loop control. Failure recovery raises success from 72% by detecting empty grasps, preventing the agent from continuing with an empty gripper, with the largest drops on hard-to-grasp objects. Arbitrary action symbols with explicit conventions nearly match semantic names, while symbols alone succeed in only 1 of 20 episodes, showing that conventions avoid the ambiguity of inferring physical effects from visual changes. Semantic action names alone remain usable but less efficient, indicating that conventions supply most grounding and semantic names act as a helpful prior. The harness handles novel objects, lighting changes, cluttered scenes, bimanual drawer opening, and letter block rearrangement without redesigning the model-facing action space.

Show-Harness in both zero-shot and fine-tuned modes consistently outperforms all baselines across task, environment, and embodiment shifts. The zero-shot agent with a frontier model achieves near-perfect success on cross-task evaluations, including held-out object combinations, while the fine-tuned small model succeeds in sim-to-real transfer using only simulated demonstrations. The same semantic interface transfers effectively between different robot arms, bridging large zero-shot models and compact fine-tuned policies. Zero-shot Show-Harness reaches perfect or near-perfect success on all cross-task evaluations, including unseen objects, far surpassing specialized VLA and code-as-policy agents. Fine-tuned Show-Harness with a small model matches the zero-shot agent on most tasks and uniquely succeeds in sim-to-real transfer, where trainable VLA baselines fail. The semantic interface generalizes across Franka and AgileX embodiments, with the zero-shot agent adapting via an interpreter and the fine-tuned policy benefiting from co-training.

The experiments evaluate a modular VLM-based robot control harness that separates perception, reasoning, and action through interchangeable plugins. Key design choices such as situated planning, action history tracking, and failure recovery substantially improve robustness by deferring uncertain decisions, preventing repetitive action loops, and detecting empty grasps. The same semantic action interface generalizes across diverse manipulation tasks, environments, and robot embodiments, enabling large zero-shot models to achieve near-perfect cross-task performance while compact fine-tuned policies uniquely succeed in sim-to-real transfer where other trainable VLA baselines fail.