One-sentence Summary

The MAI Team at Alibaba Token Foundry presents MobilePA-Bench, an interactive, stateful, and tool-centric benchmark that evaluates mobile planning agents on sub-agent collaboration, memory usage, and skill usage across 13 functional domains and 212 realistic tools, bridging GUI-centric and static function-calling evaluation paradigms while providing a foundation for agentic reinforcement learning.

Key Contributions

• MobilePA-Bench is an interactive, stateful benchmark that evaluates mobile planning agents on tool-calling and planning across 13 functional domains and 212 realistic tools within an executable sandbox.
• It provides a live simulation environment with dynamic feedback, strict tool dependencies, permission boundaries, and runtime errors, enabling rigorous assessment of an agent’s ability to adaptively recover from execution failures.
• It unifies evaluation of three advanced planning dimensions—sub-agent collaboration, memory usage, and skill usage—into realistic mobile workflows; experiments reveal that the strongest frontier model achieves only a 75.52% overall weighted score, exposing critical reliability gaps.

Introduction

LLM-driven mobile agents promise to transform devices into proactive copilots that interpret user intent, orchestrate system tools, and execute multi-step workflows. Evaluating such agents is challenging because existing benchmarks either test static function calling without real-time OS feedback or focus narrowly on GUI pixel manipulation, ignoring the system-level orchestration and error recovery needed for realistic tasks. The authors introduce MobilePA-Bench, a stateful, interactive benchmark that evaluates central planner agents across four essential dimensions—basic tool use, sub-agent collaboration, memory retrieval, and skill execution—through 1,705 tasks spanning 13 domains and 212 realistic tools, while an integrated sandbox exposes live backend mutations, permission boundaries, and runtime exceptions.

Dataset

The authors construct MobilePA-Bench, a diagnostic benchmark of 1,705 natural-language mobile tasks designed to evaluate a central planner agent. Tasks are synthesized from realistic mobile scenarios and executed inside a stateful simulation sandbox. The benchmark is organized around four capability dimensions, each verified through evidence-aligned query buckets.

Dataset composition and sources

• All tasks are synthetic but grounded in human-curated mobile workflows.
• Basic Tool Use tasks are built from human-curated seeds and inject dynamic obstacles (missing arguments, permission blocks, state mutations) to test foundational API execution.
• Memory Usage tasks are generated from coherent user-profile worlds; requests intentionally omit explicit preferences to force memory retrieval.
• Skill Usage tasks package composite multi-step routines into reusable skills; tasks require loading the correct skill and completing its tool sequence.
• Sub-agent Collaboration tasks isolate complex delegation scenarios (e.g., GUI manipulation) and record the valid downstream route and handoff payload.
• Each task is annotated with an initial sandbox state, a candidate action space, capability-specific gold targets (tool sequences, memory IDs, skill IDs, sub-agent routes), and is assigned to one of three verification buckets: Tool Call, State Change, or Agent Behavior.

Key details for each subset

• Basic Tool Use: Covers five behavioral categories (tool/parameter grounding, conditional planning, state tracking, intent revision, error recovery). Evaluated via deterministic Tool Call matching or State Change delta checks.
• Sub-agent Collaboration: Measures delegation quality rather than downstream execution. Success requires invoking the correct sub-agent and issuing a complete handoff payload (Bucket 3: Agent Behavior).
• Memory Usage: 376 tasks annotated with required gold memory IDs. A task passes only when the planner retrieves all required memory entries and satisfies the primary bucket checker.
• Skill Usage: Tasks are annotated with a gold skill ID. Success requires loading that skill and completing the downstream task. Evaluated separately under Skill-Only Routing and Mixed Tool-Skill Routing settings.

How the data is used

• The benchmark is used for zero-shot evaluation of a mobile planner agent. The planner receives a natural-language request and must decompose it into tool calls, memory lookups, skill loads, and sub-agent delegations.
• Performance is reported per dimension and aggregated into an overall score with fixed weights: Basic Tool Use 0.50, Sub-agent Collaboration 0.10, Memory Usage 0.20, Skill Usage 0.20. Missing or invalid predictions count as failures.

Processing and verification details

• A data-construction pipeline transforms mobile scenarios into executable artifacts. Dynamic obstacles are embedded into initial sandbox states to test recovery.
• Verification is evidence-aligned: Tool Call bucket checks exact tool names, call order, and argument values; State Change bucket compares terminal database transitions against annotated deltas; Agent Behavior bucket evaluates sub-agent routing and follow-up interaction quality.
• Capability-specific gates (memory retrieval, skill loading) are applied on top of the primary checker, so a task passes only when both the gate and the bucket checker succeed.

Method

The authors design MobilePA-Bench as an interactive mobile execution environment to evaluate planner agents through a closed-loop protocol. A mobile task is defined as a tuple $(q, \mathcal{S}_0, \mathcal{H}_0, \mathcal{A}_0)$(q,S0​,H0​,A0​), representing user intent, initial state, dialogue history, and candidate actions. At each interaction step $t$t, the central planner observes the query, accumulated history, and available actions to predict the next action $a_t = \pi(q, \mathcal{H}_t, \mathcal{A}_t)$at​=π(q,Ht​,At​). The executable sandbox processes this action, generating dynamic feedback $f_t$ft​ and updating the environment state. This interaction loop continues through execution feedback and error-aware replanning until the planner emits a Finish action or reaches a maximum step threshold.

As shown in the figure below:

To decouple central reasoning from low-level execution details, the system provides a unified, tool-centric action interface. The total action space consists of four categories: Direct Mobile Tools for querying or mutating system states, Sub-agent Entry Tools for routing context to specialized downstream agents, Memory Tools for retrieving persistent user profiles, and Skill Loading Tools for dynamic action-space expansion. The initial active action space $\mathcal{A}_0$A0​ is formulated by recalling top-$N$N tool schemas and available skill loaders. When the planner invokes a skill loader, the environment dynamically expands the action space for the subsequent step by binding concrete tool schemas to that skill.

The environment is grounded in a stateful mobile simulation sandbox comprising three core components. These include structured tool schemas defining API signatures, executable tool code handling validation and logic, and a shared persistent backend database maintaining live application states $\mathcal{D}_t$Dt​ and audit logs $\mathcal{O}_t$Ot​. The mobile environment state at step $t$t is modeled as $\mathcal{S}_t = \langle \mathcal{D}_t, \mathcal{O}_t \rangle$St​=⟨Dt​,Ot​⟩. When the Tool Executor processes an invocation, it applies state mutations directly to the database, records the action, and returns structured execution feedback $f_t = \langle \text{Status, ErrorType, Payload} \rangle$ft​=⟨Status, ErrorType, Payload⟩.

Refer to the framework diagram:

For evaluation, the authors employ an evidence-aligned verification protocol to address diverse completion semantics. Tasks are categorized into three distinct Query Buckets, each paired with a dedicated verification checker. Bucket 1 (Tool Call) evaluates tasks requiring deterministic operational steps by matching tool names, call order, and arguments against ground truth. Bucket 2 (State Change) applies to tasks where multiple execution paths are valid, verifying success based on the terminal database transition $\mathcal{D}_T - \mathcal{D}_0$DT​−D0​. Bucket 3 (Agent Behavior) covers open-ended tasks, evaluating the observable interaction trajectory for appropriate sub-agent delegation and user-facing interactions. Capability-specific requirements, such as memory retrieval and skill loading, are applied as additional gates on top of the primary checker.

As illustrated in the figure below:

The benchmark synthesizes tasks across four capability dimensions: Basic Tool Use, Sub-agent Collaboration, Memory Usage, and Skill Usage. These tasks are grounded in real-world mobile workflows. A scenario-labeled analysis snapshot spans 13 high-level mobile scenarios and 89 functional subcategories, providing a hierarchical distribution of the query types.

The hierarchical distribution is shown in the figure below:

Specifically for the Memory Usage dimension, tasks are synthesized from coherent user profile worlds. Requests intentionally omit explicit preferences, forcing the planner to query persistent memory. These tasks are annotated along three diagnostic axes: memory reasoning type (such as single-record grounding or conflicting-record resolution), target operation (like personalized phone actions or memory modification), and application domain.

The coverage of memory tasks is detailed in the figure below:

Experiment

The evaluation tests 13 state-of-the-art LLMs as central mobile planners across 1,705 tasks spanning four capability dimensions: basic tool use, sub-agent collaboration, memory usage, and skill usage, all within a stateful simulation sandbox that verifies success through evidence-aligned query buckets. The results show that even the strongest model achieves only 75.52% overall, with sub-agent delegation and personalized memory retrieval emerging as critical bottlenecks where performance drops sharply. No single model excels across all dimensions, revealing a gap between high-level orchestration and dependable end-to-end execution, and error analysis underscores that failures cascade across capability boundaries while planners often lack calibrated restraint when facing obstacles. These findings highlight that current frontier models remain insufficient for reliable autonomous mobile operation, and the benchmark serves as a diagnostic framework to guide improvements in compound reliability, adaptive recovery, and integrated memory grounding.

MobilePA-Bench is the only benchmark that combines a stateful, interactive mobile OS sandbox with native evaluation of sub-agent collaboration, memory usage, and skill usage, while remaining optimized for high-throughput reinforcement learning rollouts. Even the strongest frontier models achieve only 75.52% overall success, with errors cascading across capability boundaries and no single model excelling in all dimensions. The results reveal that current planners lack compound reliability, struggling most with memory-grounded personalization and delegation despite benefiting from structured skill procedures. Unlike GUI-centric benchmarks that omit advanced capabilities and static function-matching suites that lack dynamic feedback, MobilePA-Bench uniquely integrates stateful OS interaction with sub-agent, memory, and skill evaluation in a high-throughput setting. Skill usage scores are consistently higher than memory and sub-agent collaboration scores, indicating that prepackaged procedures stabilize execution but personalized retrieval and reliable delegation remain persistent weaknesses across all models.

The global tool catalog in MobilePA-Bench organizes functions into seven mobile domains, with counts ranging from 15 to 25 tools per domain. These domains cover audio and entertainment, apps and storage, display and sound, system settings, time management, AI assistant, and calls and communication, enabling diverse, realistic mobile interactions. Audio & Entertainment is the largest domain with 25 tools, while Calls & Communication has the fewest at 15 tools. The AI Assistant domain includes tools for GUI sub-agent routing and memory search, directly supporting the benchmark's Sub-agent Collaboration and Memory usage evaluation dimensions.

Across the evaluated models, overall mobile planner performance remains limited, with the strongest system reaching 75.52% and most models below 70%. Basic Tool Use is the strongest capability, while Sub-agent Collaboration and especially Memory Usage show much wider weaknesses. Capability leadership is fragmented across different models, indicating gaps in compound reliability and end-to-end execution. The top overall score is 75.52%, and seven of thirteen models remain below 70% overall. Basic Tool Use peaks at 83.85%, while Memory Usage ranges from 33.78% to 64.63%, making memory a key bottleneck. No single model leads all dimensions: leadership is split across Basic Tool Use and Skills, Sub-agent Collaboration, and Memory.

Three repeated evaluations of Qwen3.6-27B show strong run-to-run stability. Most capability scores vary by less than one percentage point, and the overall score remains within a narrow band, confirming that the benchmark yields reliable measurements despite stochastic generation. Basic Tool Use, Memory, and Skills exhibit standard deviations under one percentage point. Sub-agent Collaboration has the largest spread (2.25 points) but contributes at most 0.23 points to the overall score due to its low weight. The overall score fluctuates between 57.22% and 57.63%, an error band below 0.5 percentage points.

MobilePA-Bench evaluates mobile planners in a stateful, interactive OS sandbox that natively assesses sub-agent collaboration, memory usage, and skill usage across seven tool domains. Experiments reveal that even the strongest models achieve limited overall success, with memory-grounded personalization and delegation remaining persistent weaknesses despite the stabilizing effect of structured skills. Capability leadership is fragmented and compound reliability is lacking, while repeated evaluations confirm the benchmark's measurement stability.