One-sentence Summary

Tsinghua University and UC Berkeley introduce uni-robot-agent, a unified harness for open-weight robot-use agents that supports further training, and show that without privileged inputs Qwen3.8-Flash-Next attains a RoboDojo Score of 31.39, surpassing the published GPT-6 Astra-RoboProbe result of 28.97 and many VLA/WAM baselines while performing competitively on 19 real-world tasks; their analysis across model generations, families, and sizes links effective robot use to four general capabilities that future RUA training should develop jointly.

Key Contributions

• The paper introduces uni-robot-agent, a unified and modular harness for robot-use agents that separates shared robot tools from model runtimes and environment adapters, coordinates tool execution and feedback, and supports efficient execution and further training.
• Results show that the open-weight Qwen3.8-Flash-Next, without privileged inputs, achieves a RoboDojo Score of 31.39, surpassing the published GPT-6 Astra-RoboProbe result of 28.97 and many VLA/WAM baselines, and performs competitively on 19 real-world tasks.
• The paper provides an analysis across model generations, families, and sizes, along with privileged information, cost, and execution efficiency, and links effective robot use to four general capabilities that should be developed jointly in future robot-use agent training.

Introduction

Recent frontier models such as GPT-6 Astra and Opus 5.5 show surprising zero-shot robotic manipulation, making robot-use agents a promising way to transfer general problem-solving abilities to robotics. However, these models are proprietary and API-bound, limiting researchers to harness design and preventing fine-tuning, cheap large-scale interaction, and model-level optimization. Open-weight alternatives have lagged in spatial understanding and agentic capability, so prior efforts often relied on privileged simulator state or task-specific skills rather than systematically testing the models. The authors introduce uni-robot-agent, a general-purpose robot-use harness with unified tools and feedback, interchangeable components, and overlapping execution to make open-weight agents competitive and to support future robot-specific training.

Method

The authors introduce uni-robot-agent, a unified and modular harness designed to connect diverse agent runtimes to shared toolkits for both simulation and physical robots. The system is organized around four core principles: unified interaction, efficient execution, realistic deployment constraints, and modularity.

The modular architecture consists of five logical layers with explicit interfaces.

At the top, the agent layer manages model interaction and context, passing tool requests to the orchestration layer. The orchestration layer handles scheduling and dispatch, utilizing MCP adapters and a ToolExecutor that manages serial or parallel execution and resource exclusivity. The tool layer implements shared robot operations, such as GripperTools for motion and grasping, SceneTools for perception, SkillTools for a skill library, and EpisodeTools for episode completion. These tools access environment, perception, and action-model capabilities through the service layer, which manages runtime resources via EnvironmentService, ActorService, and PerceptionService. Finally, the backend layer supplies concrete implementations, such as EnvironmentBackend for simulation runtimes, Pi05Backend for physical robots, and Sam3Backend for inference.

To ensure unified interaction across different environments, the authors provide agents with a shared set of tools for motion, perception, skills, and episode control. These tools use common arguments, units, and feedback formats. Environment adapters translate shared tool requests into native commands.

This design allows shared tools to access observations, robot state, and primitive actions through a common interface, avoiding the need for separate tool-environment implementations. Adding a new environment only requires an adapter and its configuration.

For efficient execution, the orchestration layer combines background tool calls with resource-aware scheduling to form a pipeline between agent reasoning and tool execution.

The system supports two levels of concurrency. Between the agent and tools, background calls return a job handle, allowing the agent to continue independent reasoning. Among tool calls, operations sharing an exclusive resource execute in submission order, while parallel calls run outside that queue. This enables a two-stage pipeline where the agent can submit successive calls while previously submitted calls execute, overlapping reasoning and execution even when tools run sequentially.

The authors also emphasize realistic interaction without privileged inputs, ensuring agents rely only on information available in physical deployments, such as avoiding simulator-provided depth or task-verifier feedback. Key design choices further enhance reliability. Observations and state are kept consistent by binding images, camera calibration, and robot state to an immutable snapshot. To prevent repeated actions after communication failures, the system records request identities and execution status to reject duplicates. Motion and gripper control are coordinated to allow simultaneous action, and progress is monitored by generating intermediate targets and checking progress after each step. Action outcomes are made diagnosable by returning numerical motion feedback alongside images, and growing visual histories are managed using a sliding window that replaces old images with text placeholders.

Experiment

The experiments evaluate open-weight robot-use agents on 42 RoboDojo simulation tasks and a 19-task real-world benchmark, showing that models like Qwen3.8-Flash-Next with the uni-robot-agent harness can match or surpass fine-tuned VLA/WAM baselines and even a frontier closed model in zero-shot settings. Analysis indicates that successful robot use depends on general capabilities such as visual understanding, spatial grounding, geometric coding, and embodied reasoning, which correlate with existing foundation-model benchmarks, rather than on privileged depth inputs or a task-specific skill library. Cost and speed results further show that open-weight models offer a favorable performance-to-cost trade-off, and pipelining model inference with physical execution substantially reduces episode duration. Overall, the findings support the viability of open-source robot-use agents when paired with an appropriately designed harness.

On RoboDojo, open-weight models using the uni-robot-agent harness outperform the listed VLA and WAM baselines on average success rate and score, with one open-weight model also surpassing a GPT-6 Astra configuration that uses RoboProbe. The uni-robot-agent harness appears important, as replacing it on the strongest baseline substantially reduces performance. GPT-6 Astra with uni-robot-agent achieves the best overall score, while open-weight models offer a much more cost-efficient performance trade-off. Open-weight models with uni-robot-agent surpass the VLA and WAM baselines on average, and Qwen also edges out GPT-6 Astra paired with RoboProbe. Robot-use agents tend to score higher on memory and open-vocabulary capabilities than VLA and WAM baselines, but the open-weight agents underperform on long-horizon tasks relative to Xiaomi-Robotics-1 and OpenWAM. Harness choice has a large impact: moving the strongest baseline away from uni-robot-agent causes a notable drop in both success rate and score. GPT-6 Astra with uni-robot-agent reaches the best performance, but open-weight models provide a substantially cheaper route to comparable scores.

In real-world evaluation, uni-robot-agent with Qwen3.8-Flash-Next matches the performance of fine-tuned π0.5 and outperforms zero-shot Dream-1, while with GPT-6 Astra it leads all baselines by a large margin. The approach tends to perform better on compositional tasks and unseen tasks, whereas VLA and WAM baselines are relatively stronger on object generalization. These results support applying uni-robot-agent to real-world robot tasks. Uni-robot-agent with Qwen3.8-Flash-Next matches the performance of fine-tuned π0.5 and exceeds zero-shot Dream-1 in real-world evaluation. Uni-robot-agent with GPT-6 Astra outperforms all other baselines by a large margin. Robot-use agents perform better on compositional and unseen tasks, while VLA and WAM models perform well on object generalization.

The RoboDojo benchmark evaluates robot-use agents using a uni-robot-agent harness against VLA and WAM baselines, showing that the harness substantially improves performance and that open-weight models with it outperform the baselines on average. GPT-6 Astra with the harness achieves the best overall score, while open-weight models provide a much more cost-efficient route to comparable results. In real-world evaluation, the approach matches or surpasses fine-tuned and zero-shot baselines, with the largest gains on compositional and unseen tasks, whereas VLA and WAM models remain relatively stronger on object generalization.