One-sentence Summary

Researchers from Harbin Institute of Technology, Shenzhen Loop Area Institute, and AI Training Platform propose KnowAct-GUIClaw, a Know-Route-Act-Reflect framework with a self-evolving memory and skill library that overcomes OpenClaw’s cross-platform GUI automation limitations, achieving $64.1\%$64.1% on the long-horizon MobileWorld benchmark with Kimi-2.6—surpassing all agential frameworks and closed-source models like Seed-2.0-Pro and GPT-5.5—and enabling transferable knowledge and skill gains of $8.5\%$8.5% with Kimi-2.6 and $16.2\%$16.2% with Qwen3.5-35B-A3B.

Key Contributions

• The KnowAct-GUIClaw framework decouples task orchestration and visual manipulation via a Know-Route-Act-Reflect pipeline, where a host agent plans long-horizon tasks and a pluggable GUI subagent executes across Android, iOS, HarmonyOS, and Windows.
• The system incorporates attribution-aware persistent experience memory and state-validated self-evolving skill libraries that accumulate user feedback and verified routines, enabling continuous improvement of task decomposition and tool calls without weight updates.
• Extensive experiments show the Kimi-2.6-powered GUIClaw achieves 64.1% on the MobileWorld benchmark, outperforming all prior agentic frameworks and closed-source models such as Seed-2.0-Pro and GPT-5.5; the memory and skill modules transfer across base models, yielding gains of +8.5% on Kimi-2.6 and +16.2% on Qwen3.5-35B-A3B.

Introduction

LLM-based personal assistants are evolving beyond single-turn dialogue to manage persistent, cross-application tasks. Many real-world requests, however, require interacting with graphical user interfaces where APIs are unavailable, and simply grafting a standalone GUI agent onto a host framework leads to brittle behavior: free-text summaries lose intermediate data across app transitions, partial screen observations hide critical context, prior task trajectories are discarded instead of reused, and faster non-visual shortcuts remain unexploited. The authors introduce KNOWACT-GUICLAW, a framework that pairs a high-level orchestrating host with a lightweight GUI executor. It grounds decisions in retrievable structured memory and self-evolving skill libraries, enabling the system to plan across apps, resume stalled tasks, validate shortcuts against real-time screen state, and continuously distill successful traces into reusable knowledge for future runs.

Method

The authors propose KNOWACT-GUICLAW, a host-centric multi-agent framework that organizes long-horizon mobile GUI task execution into a four-stage loop: Know, Route, Act, and Reflect. This architecture separates high-level task management from low-level device interaction, pairing a central orchestrator with a specialized GUI subagent. The host agent retains responsibility for user-facing task orchestration and long-horizon context maintenance, while the GUI subagent operates as a self-contained engine dedicated to low-level device control.

In the Know stage, the system gathers context actively before any GUI action. The host agent maintains the running session context and recalls stored memories, such as session history and user profiles, only when relevant. Prior GUI memories and candidate skills are retrieved by semantic similarity and kept advisory, never overriding the current instruction, while policy memory is injected directly. This advisory role allows the system to correct app and route selection based on past experiences without replacing live screen evidence.

The Route stage handles task decomposition and information contracts. The routing policy emits either a single GUI task or an ordered multi-app workflow. In the multi-app case, each subtask is defined as a goal-level tuple $(g_i, h_i, I_i, O_i)$(gi​,hi​,Ii​,Oi​), where $g_i$gi​ is the app-scoped goal, $h_i$hi​ optionally narrows the app, $I_i$Ii​ names the required inputs, and $O_i$Oi​ names the values to return. To manage cross-app data flow, the system employs a short-lived blackboard $B$B. A subtask sees only the declared inputs already on $B_{i-1}$Bi−1​, and only its declared outputs are written back from the trajectory evidence: $G(g_i, h_i, B_{i-1}[I_i]) \rightarrow \tau_i, \quad E(\tau_i, O_i) \rightarrow B_i[O_i].$G(gi​,hi​,Bi−1​[Ii​])→τi​,E(τi​,Oi​)→Bi​[Oi​]. If a required input or declared output is missing, the workflow fails closed rather than running on incomplete state.

This blackboard-mediated execution ensures that cross-app tasks pass only declared outputs, reducing history preservation requirements for each GUI task and keeping later goals tied to explicit inputs.

During the Act stage, the GUI subagent executes each routed subtask over a hybrid action space $\mathcal{A} = \mathcal{A}_{\text{gui}} \cup \mathcal{A}_{\text{skill}} \cup \mathcal{A}_{\text{shortcut}} \cup \mathcal{A}_{\text{ask}}$A=Agui​∪Askill​∪Ashortcut​∪Aask​. This space includes human-like GUI primitives, distilled skills, Android deeplinks and intents that bypass lengthy navigation, and intervention actions for user input. The executor runs a grounded observe-reason-act loop: $p_t \rightarrow o_t \rightarrow a_t \rightarrow e_t \rightarrow o_{t+1},$pt​→ot​→at​→et​→ot+1​, where $p_t$pt​ is the prompt carrying the subtask and blackboard state, $o_t$ot​ is the observation, $a_t$at​ is the action, and $e_t$et​ is the result. Skills and shortcuts are validated against the current state before use to balance efficiency and reliability.

Finally, the Reflect stage performs trajectory distillation and skill evolution. After each GUI task, reflection condenses the trajectory into a short note for checkpointing. For selected trajectories, the system extracts reusable procedures rather than storing raw actions. A vision LLM rewrites GUI events into structured evidence to generate candidate skills, which are normalized and checked for supported actions and valid-state coverage. The system separates repair from new extraction; when a reused skill fails, reflection updates that skill in place by narrowing descriptions or refreshing stale targets. Additionally, experience memory stores textual policies derived from running trajectories, using separate success and failure prompts to induce actionable memory items that guide future routing and GUI execution. Shortcut candidates mined from app manifests undergo validation runs to ensure usability and page status before being promoted to one-step skills.

Experiment

KNOWACT-GUICLAW is evaluated on MobileWorld and AndroidDaily, where it decomposes long-horizon tasks via a host–router, experience memory, and executable skills. Ablations show that host coordination and memory raise success rates with modest token overhead, skills reduce steps and tokens, and both memory and skills transfer across model families. Case studies illustrate how routing, memory correction, skill compression, and host recovery enable robust cross-app workflows, while cross-platform checks on HarmonyOS and Windows highlight remaining grounding and control challenges.

KNOWACT-GUICLAW combines a host model with experience memory and skills to substantially improve GUI task success over a base executor. The best configuration, using Kimi-K2.6 as both host and executor with skills enabled, achieves a 64.1% success rate, exceeding all listed general models. Knowledge distilled from Kimi-K2.6 trajectories transfers to a Qwen executor, lifting its success from 24.8% to 41.0% and demonstrating cross-model transferability. Adding a host with experience memory and skills consistently raises success rate; enabling skills also reduces the number of GUI steps and total tokens. With Kimi-K2.6 as the host and executor, KNOWACT-GUICLAW reaches 64.1% success, outperforming general models like GPT-5.5 and Gemini-3.1-Pro on this benchmark. Experience memory and skills distilled from Kimi-K2.6 trajectories transfer to a Qwen 35B executor, boosting its success rate by 16.2 points over the base executor alone.

Adding host coordination and experience memory raises success rate on MobileWorld GUI-Only tasks for both a 35B and a 397B executor, while total token consumption stays nearly flat or increases only modestly because the gain comes from better task decomposition rather than more GUI observations. Enabling skills further reduces GUI steps and total tokens while success rate still improves, with larger relative savings when the strong host resolves subtasks directly. Experience memory and skills can transfer across model families, as a 35B executor using Kimi‑distilled memory and skills reaches higher accuracy than its native augmented configuration. Host and memory boost success rate at both executor scales, from 24.8 to 34.5 for the smaller model and from 40.7 to 43.3 for the larger one, while total tokens remain essentially unchanged for the small executor and increase by about 7% for the large executor. Skills reduce GUI steps and total tokens; for the 397B host, steps fall from 26.8 to 23.7 and total tokens drop by roughly 5%, yet success rate still rises. Host overhead is much lower with the 397B host (under 4% of total tokens) than with the 35B host (about 23%), because the strong host can handle subtasks directly instead of delegating every one to the weaker executor. Cross-model transfer of memory and skills is effective: a 35B executor using Kimi-K2.6 distilled memory and skills achieves a 41.0 success rate, outperforming the 37.9 of its own host–memory–skills setup.

On skill-invoking tasks, enabling skills reduces GUI steps by about three steps per task and cuts total tokens by roughly 6% for both executor sizes, while single-run success rate improves. The efficiency gain comes from shorter trajectories and fewer screenshot observations, and the success-rate benefit is larger for the smaller executor. Skills consistently shrink GUI steps by 3.3 per task and total tokens by approximately 6% across both the 35B and 397B executors. The 35B executor gains 4.9 percentage points in single-run success rate from skills, compared to 1.9 points for the 397B executor, while pass@3 stays nearly flat.

On the AndroidDaily benchmark, the proposed model achieves the highest end-to-end success overall, with its strongest advantages over GUI-only baselines appearing on Analyze tasks and complex (Comp.) tasks. The performance gap between the resolved and all evaluations reveals that app unavailability and environment mismatch account for a meaningful portion of failures, rather than policy quality alone. The largest margin over baselines occurs on Analyze tasks, where the resolved model reaches 81.25% versus 61.41% for the strongest baseline. Complex (Comp.) tasks show a similarly wide lead, with the resolved model at 75.00% compared to baseline scores around 60–62%. Scoring unavailable tasks as zero (All setting) reduces total success from 80.56% to 68.40%, highlighting the impact of app and environment gaps on end-to-end reliability.

From configuration A to F, single-run success rate rises steadily from 24.8% to 46.2%, while allowing any of three repeated attempts lifts the upper bound to 59.8% in the best configuration. The stricter all-three-success metric increases from 15.4% to 32.5%, confirming that improvements are consistent and not due to large random volatility. Taking any of three runs nearly doubles the chance of solving a task compared to the single-run rate, reaching 59.8% in the highest-performing configuration. Requiring all three runs to succeed yields lower but consistently improving rates (15.4% to 32.5%), showing that progress in control reliability does not come with erratic fluctuations.

The experiments evaluate a GUI task execution framework that augments a base executor with a coordinating host model, experience memory, and reusable skills on MobileWorld and AndroidDaily benchmarks. Adding host coordination and memory consistently raises success rates across executor scales with minimal token overhead, while enabling skills further reduces GUI steps and total tokens, especially for weaker executors. Cross-model transfer of distilled memory and skills proves effective, allowing a smaller executor to exceed its own native augmented configuration. End-to-end deployment analysis reveals that app unavailability and environment mismatches cause a significant share of failures, and multi-attempt reliability checks confirm that improvements are consistent rather than erratic.