One-sentence Summary

HexaFuture.ai's HexaShell is a thin robot-coding harness that lets a VLM agent control a dual-arm robot through one command line, fix failed attempts by writing code tools, and locate objects via camera-ray triangulation using RGB views, achieving 39.7% success and 48.0 progress on all 42 RoboDojo tasks and setting a new state of the art.

Key Contributions

• HexaShell is a thin robot-control harness that lets an unchanged coding agent drive a two-arm robot through one command line and turns failed-attempt logs into reusable, repairable tool programs.
• The tools use only RGB observations and calibrated multi-view geometry, including two-camera ray intersection, ray-plane intersection, and active wrist-camera movement, to locate 3D object positions relative to each other and to the gripper.
• Tools are built on ten layouts of one seed per task, frozen, and tested on ten new layouts of the same seed; on all 42 RoboDojo tasks, HexaShell reaches 39.7% success and a 48.0 progress score, a new state of the art. The analysis identifies contact rather than reasoning as the main limit, with 92.5% success on pushing and 8.3% on tight insertion, and the evolved tools are open sourced.

Introduction

Robot manipulation leaderboards are now led by learned policies, including vision-language-action and world-action models that map large demonstration datasets directly to robot actions, while VLM-based agents are beginning to compete through fixed harnesses. Coding agents offer an alternative because they can read, run, and repair code and keep the fix, but prior robot coding work mostly generated per-instruction programs over human-written primitives; on RoboDojo, a VLM without its own tools reaches only 22.5% through a fixed harness. A core limitation is the pixel-to-position gap: VLM agents judge metric distances poorly, and monocular depth networks can be off by tens of millimeters, too coarse for reliable grasping. The authors introduce HexaShell, a thin harness for an unchanged coding agent that learns persistent robot tools with unit tests and repairs them after failures, using calibrated multi-view triangulation and active perception to turn selected pixels into world positions. On new layouts of all 42 RoboDojo tasks, HexaShell reaches 39.7% success and a 48.0 progress score, a new state of the art.

Method

The authors design HexaShell to enable coding agents to solve robot manipulation tasks using only RGB observations. The architecture comprises three core components: a harness that translates commands into motion, a set of tools that measure 3D positions from RGB images, and an evolution loop that iteratively improves these tools based on task failures.

As shown in the figure below:

The harness isolates the acting agent within a sandboxed container that lacks direct network access, API keys, or host directories. The agent interacts with the environment through a single executable and a directory of observations refreshed on request. These observations include three RGB images from head and wrist cameras, along with camera intrinsics, poses, tool center point poses, and joint angles. The agent issues base commands such as move, rotate, point, gripper, and obs. Behind the command line, an egress proxy routes traffic to a harness server. This server validates each command, interpolates a straight Cartesian path, solves inverse kinematics per waypoint using cuRobo, and times the joint path with a trapezoidal profile under the official robot limits. The timed path is sampled at $25$25 Hz and sent to the simulator. Crucially, the harness contains no motion planner or collision model beyond the robot and the table, and success is judged solely by the unmodified benchmark checker.

To bridge the gap between 2D pixels and 3D world coordinates without depth sensors, the system relies on geometric constraints derived from calibrated cameras. A pixel and its calibration define a ray in the world frame, but a ray alone is insufficient for positioning. The tools leverage three types of constraints using standard OpenCV geometry. The first is the table plane, measured once per episode by a shared tool that sweeps candidate heights and warps images to find the height where camera views agree best. The second constraint is any other horizontal plane with a known height, such as the top face of a block. The third constraint utilizes a second camera view for triangulation. If two cameras observe the same point, their rays intersect to yield the 3D position.

As shown in the figure below:

When a single view is inadequate, such as when an object is partially occluded or viewed from a poor angle, the robot must move a wrist camera to obtain a second perspective. This movement consumes action steps, representing the cost of measuring without depth. For instance, in tasks requiring precise localization of a held object, the system matches features across two current camera views to triangulate the center, filtering candidates by radius and failing gracefully if no clear match is found.

Tools are implemented as directories containing a Python file and an interface description. The Python file declares subcommands and a run function over an EpisodeAPI, which provides joint angles, TCP poses, RGB images, and motion primitives. Measurement commands that do not move the robot are free, while commands that move an arm consume steps from the task limit. A shared pool of tools remains unmodified by the optimization process, while all other tools are specific to the task.

The evolution loop drives the continuous improvement of these tools. A run begins on a specific task and layout. The acting agent attempts the layout with the current tool set. Upon success, an optimizer agent reads the trajectory and logs which tools worked. On failure, the optimizer analyzes the command log, camera frames, and current tool code to diagnose the issue. It then writes a diagnosis and modifies or creates a tool to address the failure. A delivery check ensures that the modified files are complete, loadable, and do not reference task specific objects in the interface text. If the check fails, the edit is reverted. The simulator restarts, and the same layout is attempted again, allowing up to five edits per layout.

As shown in the figure below:

After processing ten layouts, the optimizer consolidates its notes, the tool set is frozen, and it is retested on development and new layouts. This iterative process allows the system to refine complex behaviors, such as adjusting motion schedules for moving objects or relaxing overly strict clearance constraints based on unverified monocular rays, ultimately enabling the agent to recover from failures and complete challenging manipulation sequences.

Experiment

This work evaluates a dual-arm RGB-only agent that builds geometric measurement tools on RoboDojo’s 42 simulated tasks, comparing it against leaderboard policies, depth-network variants, and clean versus cluttered scenes. The results show that these tools give the agent strong performance in memory and open tasks, preserve success when distractors are added, and benefit from active second views, while precise insertion, deformable material, and articulated contact remain the main weaknesses. Case studies and failure analysis indicate that the optimizer can repair local measurement, timing, grasp, and early-stop errors, but it does not overcome the fundamental contact-level limitations of tasks requiring tight insertion or accurate pouring without depth.

Triangulation outperforms the depth network on overall leaderboard score and in most task categories, especially precision, memory, and generalization; the open category is close, with a small depth advantage. The depth network's median errors at the head and wrist cameras are large relative to the centimeter accuracy needed for grasping and placement, while triangulation does not have the same fixed depth-estimation floor when geometric assumptions hold. Rendered depth remains higher but is reference-only. Across paired layouts, triangulation converts more depth-network failures into successes than it loses, with the largest gains in push T, imitate sorting sequence, and play Xylophone. Losses relative to the depth network are smaller and concentrated in round-container tasks such as pouring and stacking.

The triangulation-based agent spends more executed steps on viewing moves than the depth-network agent, and a much larger share of episodes include at least one viewing move. The extra viewing is not associated with a tighter step budget overall: the agent succeeds more often, while slightly fewer failures approach the step limit and fewer episodes end at the limit. Viewing cost is concentrated in tasks where the robot must measure features that earlier tools treated as fixed. Viewing moves are much more common with triangulation, occurring in 43% of episodes compared with 10% for the depth network. The triangulation agent achieves more successes overall despite using a higher median fraction of the step limit, and it has slightly fewer failures that consume at least three quarters of the limit.

A capability map shows tool-based success varies strongly by physical interaction class. The depth tools improve pushing and rigid manipulation the most, while tight insertion and articulated objects remain difficult. Failure patterns also differ: deformable, articulated, and rigid failures mostly exhaust the budget, whereas memory tasks often end in false completion and moving objects often lead to giving up. The triangulation tool reaches 92.5% success on pushing and 50.0% on rigid manipulation, while tight insertion and articulated tool use stay below 10%. The learned policy has the opposite profile, doing better on insertion and deformable material but weaker on pushing. Budget exhaustion dominates failures in deformable, articulated, and rigid classes, while declared completion is most common for memory tasks.

Across seven recovered failures, the optimizer made small, localized edits to one tool that fixed measurement, timing, grasp selection, or premature fallback behavior. Most recoveries relied on command logs, tool feedback, and code rather than visual inspection, and the next attempt succeeded in all seven cases. Frozen retests showed strong generalization for some tasks but weaker task-level robustness in harder classes such as cloth folding and tube insertion. The optimizer mostly worked from textual and numerical evidence such as command logs and refusal reasons, with image analysis needed in only a minority of cases. Retest generalization was uneven: some recoveries fully generalized across development layouts, while cloth folding and tube insertion remained fragile and had lower task-level success.

These experiments evaluate a triangulation-based tool agent against a depth-network baseline on manipulation benchmarks, examine its step and viewing costs, break down performance by physical interaction class, and test an automated optimizer for recovering failures. Triangulation improves overall leaderboard score and succeeds more often in precision, memory, and generalization tasks, although it uses more viewing moves and does not fix difficult insertion or articulated-tool cases. The capability analysis shows depth tools mainly help pushing and rigid manipulation while tight insertion and deformable tasks remain challenging, with budget exhaustion as the dominant failure mode. The optimizer makes small localized fixes that usually succeed immediately, but retest generalization is uneven, especially for fragile tasks such as cloth folding and tube insertion.