One-sentence Summary

The authors from HKU MMLab and Meituan introduce UniClawBench, a capability-driven benchmark that evaluates proactive agents in dynamic real-world settings through 400 bilingual tasks grounded in five foundational capabilities—Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination—using live Docker containers and a closed-loop evaluation strategy to disentangle base model capabilities from agent framework design.

Key Contributions

• UniClawBench introduces a capability-driven benchmark that organizes 400 bilingual real-world tasks around five foundational model capabilities (Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination), replacing scenario-based taxonomies that conflate multiple skills.
• A closed-loop evaluation strategy runs agents in live Docker containers with fine-grained step-by-step completion checkpoints and a three-role framework (executor, hidden supervisor, and user simulator) to simulate realistic multi-turn human feedback without exposing grading criteria.
• Evaluations across multiple agent frameworks (OpenClaw, Nanobot, EDICT) and state-of-the-art models disentangle intrinsic model capabilities from framework-level design choices, revealing how both jointly determine real-world performance.

Introduction

Large language models have evolved into proactive agents that autonomously control browsers and terminals to serve as always-on digital assistants, yet existing benchmarks fail to capture their real-world performance. They rely on sandboxed environments with static answers, evaluate only single-turn interactions, and organize tasks by scenario rather than by the underlying capabilities required, making it impossible to pinpoint why an agent fails. The authors introduce UniClawBench, a capability-driven benchmark that tests agents across five foundational dimensions (Skill Usage, Exploration, Long Context, Multimodal, and Cross-Platform) using 400 bilingual real-world tasks in live Docker containers. To handle the instability of dynamic environments and the risk of leaking evaluation criteria during multi-turn feedback, they design a three-role closed-loop evaluation with a hidden supervisor and a user simulator that provides natural corrections without exposing ground-truth answers.

Dataset

The authors introduce UniClawBench, a manually constructed benchmark of 400 bilingual tasks designed to evaluate agent capabilities across five core dimensions. Each task is built from genuine day-to-day user scenarios and packaged with all necessary context, tools, and hidden evaluation references.

Dataset composition and sources

• Total tasks: 400, evenly split between English (200) and Chinese (200).
• All tasks are hand-crafted by the authors based on realistic personal workflows, not scraped from existing datasets.
• Each task is assigned to one of five capability categories according to its primary bottleneck.

Key details for each subset

• Skill Usage (80 tasks, 40 EN / 40 CN): Requires selecting, inspecting, and operating specific tools or APIs (e.g., OCR, spreadsheets, SQLite, Git, document conversion). Success depends on correctly using the provided tool interface to produce a verifiable artifact.
• Exploration (80 tasks, 40 EN / 40 CN): Tests open-ended investigation under incomplete or misleading information. Agents must reverse-engineer undocumented APIs, audit configurations, trace data to authoritative sources, or solve puzzles. Outputs include both a final artifact and supporting audit records.
• Long-Context Reasoning (80 tasks, 40 EN / 40 CN): Evaluates synthesis across many heterogeneous sources (web pages, videos, PDFs, emails). The challenge is maintaining global consistency and reconciling conflicts, not merely retrieving isolated facts.
• Multimodal Understanding (80 tasks, 40 EN / 40 CN): Grounded in images, videos, and audio. Tasks require extracting non-textual evidence to recreate figures, generate subtitles, or organize visual collections. Correctness depends on actual visual/audio content, not metadata.
• Cross-Platform Coordination (80 tasks, 40 EN / 40 CN): Involves synchronizing state and transferring information across web, desktop GUI, files, calendars, and other applications. Agents must use real desktop tools and preserve evidence across platform boundaries.

Task package schema Each task is a complete package containing:

• User instruction
• Task-specific context
• Input files, web resources, tools, services, and skills
• Expected output formats (e.g., JSON, CSV, office documents, diagrams, GUI states)
• Hidden evaluation references for automated scoring

No cropping or filtering is applied; tasks are designed from scratch so that the primary capability bottleneck is clear while allowing natural auxiliary operations.

How the benchmark is used UniClawBench is employed in a three-role closed-loop evaluation: an executor agent attempts the task, a hidden supervisor agent monitors and scores, and a public user simulator provides multi-turn feedback without leaking private grading information. The entire pipeline runs inside a Docker-based environment with task-resource injection, artifact collection, and a supervisor-driven scoring system. The benchmark is not used for training; it serves solely to measure agent proficiency across the five capability dimensions.

Method

The authors design UniClawBench around a capability-oriented task taxonomy that organizes evaluation scenarios into a hierarchical task-package structure. This taxonomy ensures that each task is categorized by the specific agent capability it exercises, enabling fine-grained analysis of model performance across diverse tool-use and interaction patterns. The benchmark composition is then instantiated through a collection of task packages, each containing problem definitions, initial states, and expected outcomes, with comprehensive statistics covering task distribution and difficulty.

To evaluate agents in a multi-turn, interactive setting without leaking private grading information, the framework adopts a three-role closed-loop strategy. The evaluated executor agent interacts with a public user simulator that issues instructions and provides conversational feedback, while a hidden supervisor agent monitors the execution, assesses correctness, and manages scoring. This decoupling prevents the executor from accessing ground-truth evaluation signals, preserving the realism of the interaction and ensuring that the agent must rely solely on user-facing feedback.

The execution and evaluation pipeline is containerized using Docker to guarantee reproducibility and isolation. Task resources are injected into the runtime environment, and the agent’s actions are executed within this sandbox. After the interaction completes, an artifact collection module gathers logs, tool outputs, and final states. A supervisor-driven scoring pipeline then processes these artifacts according to task-specific rubrics, producing structured evaluation results. This system design supports scalable, consistent assessment of agent performance across the entire benchmark.

Experiment

The evaluation setup uses a capability-oriented benchmark of 400 real-world tasks across five dimensions, assessed through a closed-loop multi-turn interaction where a hidden supervisor grades progress and a simulated user provides natural feedback without exposing evaluation criteria. Cross-model experiments show that even the strongest models achieve only partial progress and fail to complete most tasks, with the largest gaps in long-context reasoning, multimodal understanding, and cross-platform coordination. Cross-framework comparisons reveal that framework design profoundly impacts success, with a unified trajectory approach outperforming multi-agent orchestration or extreme token efficiency, and multi-turn user feedback consistently improves performance but does not overcome the fundamental reliability challenges of extended real-world task execution.

Under the OpenClaw framework, all evaluated models achieve overall pass rates well below 50%, underscoring the benchmark's extreme difficulty. Open-source models like Kimi-2.6 and Qwen-3.5-Plus lead the field, outperforming several closed-source counterparts. A consistent gap between high average checkpoint scores and low final pass rates reveals a widespread 'halfway failure' pattern where models make partial progress but rarely complete tasks successfully. Kimi-2.6 and Qwen-3.5-Plus achieve the highest overall pass rates among the tested models, surpassing closed-source alternatives. All models struggle most with Multimodal and Cross Platform tasks, where pass rates are near zero, while Exploration tasks see the highest success rates. Every model exhibits a large discrepancy between average checkpoint scores and final pass rates, indicating that partial progress rarely translates into fully verified task completion.

Framework design decisively shapes agent performance. OpenClaw's centralized architecture achieves the highest pass rates by preserving full task context, while EDICT's multi-agent setup yields high average scores but suffers from coordination breakdowns that prevent task completion. Nanobot offers extreme token efficiency but often fails to produce complete solutions required for strict pass criteria. OpenClaw attains the top overall pass rate for both models, with Claude Opus-4.8 reaching 0.475 compared to 0.415 for EDICT and 0.385 for Nanobot. EDICT shows the largest gap between average score and pass rate, indicating that tasks frequently reach partial progress but fail final verification due to multi-agent handoff friction.

The OpenClaw benchmark proves extremely challenging, with all models achieving low overall pass rates and a pervasive halfway failure pattern where partial progress seldom translates into completed tasks. Open-source models such as Kimi-2.6 and Qwen-3.5-Plus outperform several closed-source alternatives, and a centralized framework design (OpenClaw) surpasses multi-agent (EDICT) and token-efficient (Nanobot) approaches by maintaining task context and avoiding coordination breakdowns. Multimodal and cross-platform tasks remain near-impossible for all systems, while exploration tasks see the highest success rates.