One-sentence Summary

Researchers from Tsinghua University, Xiaomi, et al. construct the Uni-GUI cross-platform GUI interaction dataset and propose UI-MOPD, the first method that uses multi-teacher on-policy distillation to continually learn GUI agents, dynamically selecting platform-specific teachers to transfer behavioral priors to a shared policy, thereby enabling adaptation to new platforms while preserving existing capabilities, and achieving 38.2% and 12.0% task success rates on OSWorld and MobileWorld, respectively.

Key Contributions

• Constructs Uni-GUI, a high-quality cross-platform GUI interaction dataset providing executable trajectories across desktop and mobile environments, establishing a data foundation for multi-platform agent training.
• Proposes UI-MOPD, the first method to introduce multi-teacher on-policy distillation into continual learning for GUI agents, dynamically routing rollouts to platform-specific teachers and using platform-conditioned distillation to transfer behavioral priors to a shared policy, enabling adaptation to new platforms while preserving existing capabilities.
• Experiments on OSWorld and MobileWorld show UI-MOPD achieves task success rates of 38.2% and 12.0%, respectively, substantially outperforming baselines such as model merging and distillation from static offline trajectories, and demonstrating effective balance between cross-platform capability retention and new-platform adaptation.

Introduction

The authors address the challenge of building GUI agents that can operate across heterogeneous platforms such as desktop and mobile while retaining the distinct interaction conventions of each. Real-world tasks often span multiple interface types, but prior work treats cross-platform learning as a simple aggregation of mixed training data, which risks averaging out platform-specific behaviors and causing catastrophic forgetting in continual learning settings. High-quality cross-platform trajectories are scarce, and directly combining conflicting action semantics (e.g., closing a window versus pressing a back button) degrades agent performance. To overcome these limitations, the authors introduce Uni-GUI, a carefully filtered dataset of roughly 10,000 high-quality cross-platform trajectories collected through a unified data harness. They then propose UI-MOPD, the first method to apply multi-teacher on-policy distillation to GUI agent training. By dynamically routing rollouts to platform-specific teachers, UI-MOPD provides stable behavioral anchors that preserve native interaction patterns while allowing a shared policy to adapt to new platforms, achieving substantial gains on both OSWorld and MobileWorld without sacrificing previously learned capabilities.

Method

The authors propose UI-MOPD, a two-stage training pipeline that produces a single, unified GUI agent capable of operating across desktop and mobile platforms while retaining each platform’s distinct interaction patterns. The first stage leverages supervised fine‑tuning (SFT) of a vision‑language foundation model on high‑quality trajectories from each platform. This yields two separate expert teachers: a desktop teacher $\pi_{\mathrm{ref}}^d$πrefd​ and a mobile teacher $\pi_{\mathrm{ref}}^m$πrefm​. In the second stage, a multi‑teacher on‑policy distillation (MOPD) procedure integrates the specialized knowledge of both teachers into a shared student policy $\pi_\theta$πθ​ through online reinforcement learning.

The core of MOPD lies in formulating multi‑teacher knowledge transfer as a conditional behavioral constraint during online policy optimization. Instead of merging model weights or distilling from static offline data, the student policy $\pi_\theta$πθ​ generates rollouts from its current policy, and teacher supervision is only applied to the states actually visited by the student. This concentrates the distillation signal on the student’s own decision distribution, making it better aligned with the student’s current errors. Teacher supervision is routed by platform: each rollout is paired with its corresponding platform‑specific teacher, and the resulting KL penalty acts as a platform‑conditioned behavioral anchor.

The per‑token on‑policy KL divergence between the student and the routed teacher is defined as

$D_{\mathrm{KL}}^{(t,i)} = D_{\mathrm{KL}}\left(\pi_\theta(\cdot \mid h_t^{(i)}) \;\|\; \pi_{\mathrm{ref}}^{(i)}(\cdot \mid h_t^{(i)})\right),$DKL(t,i)​=DKL​(πθ​(⋅∣ht(i)​)∥πref(i)​(⋅∣ht(i)​)),

where $h_t^{(i)}$ht(i)​ is the decoding state at token $t$t for sample $i$i. To avoid computing the full KL over the entire vocabulary, the method employs the K3 estimator, which requires only the student and teacher log‑probabilities of the sampled token. Defining

$\delta_t^{(i)} = \log \pi_{\mathrm{ref}}^{(i)}(y_t \mid h_t^{(i)}) - \log \pi_\theta(y_t \mid h_t^{(i)}), \quad \rho_t^{(i)} = \exp(\delta_t^{(i)}),$δt(i)​=logπref(i)​(yt​∣ht(i)​)−logπθ​(yt​∣ht(i)​),ρt(i)​=exp(δt(i)​),

the token‑level estimate is

$\hat{D}_{\mathrm{KL}}^{(t,i)} = \rho_t^{(i)} - \delta_t^{(i)} - 1.$D^KL(t,i)​=ρt(i)​−δt(i)​−1.

This estimator is nonnegative, unbiased under samples from $\pi_\theta$πθ​, and empirically exhibits lower variance than direct log‑ratio approaches. The mini‑batch MOPD loss aggregates these token‑level estimates, weighted by an adaptive mask:

$\mathcal{L}_{\mathrm{MOPD}}(\theta) = \frac{\sum_{i \in B} \sum_{t} m_t^{(i)} \mu^{(i)} \hat{D}_{\mathrm{KL}}^{(t,i)}}{\sum_{i \in B} \sum_{t} m_t^{(i)} \mu^{(i)}}.$LMOPD​(θ)=∑i∈B​∑t​mt(i)​μ(i)∑i∈B​∑t​mt(i)​μ(i)D^KL(t,i)​​.

Here $m_t^{(i)}$mt(i)​ indicates the token’s loss mask (e.g., masking out padding or system tokens), and $\mu^{(i)}$μ(i) is a group‑level adaptive mask that disables the KL penalty when the average reward of a prompt group already exceeds a threshold $\tau_{\mathrm{KL}}$τKL​. This adaptive masking prevents the teacher constraint from stifling exploration when task feedback is already strong, while preserving behavioral guidance on low‑reward rollouts.

Platform‑conditioned teacher routing is implemented by assigning each rollout to its platform‑specific teacher based on a data source label $s_i$si​ recorded during construction:

$\pi_{\mathrm{ref}}^{(i)} = \begin{cases} \pi_{\mathrm{ref}}^m, & s_i \in \mathcal{S}_{\text{mobile}}, \\ \pi_{\mathrm{ref}}^d, & s_i \in \mathcal{S}_{\text{desktop}}. \end{cases}$πref(i)​={πrefm​,πrefd​,​si​∈Smobile​,si​∈Sdesktop​.​

The student itself remains a single shared policy, and no additional models or agents are required at inference time. During each RL update, the student samples rollouts from mixed‑platform prompts, the mini‑batch is partitioned by platform, teacher log‑probabilities are computed, and the per‑platform signals are merged back to compute the final loss. This design ensures that desktop‑specific behaviors (mouse, keyboard, window switching) and mobile‑specific behaviors (tap, swipe, navigation) are preserved as distinct anchors within the shared parameter space.

The reward signal is a structured outcome reward computed from the agent’s action output, which is a JSON containing action type, coordinates, text, and other typed fields. For each action type, a set of required dimensions is defined (e.g., action type correctness, coordinate containment in a target bounding box, scroll direction, key set equality, case‑insensitive text matching). The fraction of matched dimensions $f_a$fa​ determines the reward:

$R(x, y) = \begin{cases} 1.0, & f_a = 1, \\ -0.5, & 0 \le f_a < 1, \\ -1.0, & \text{unparsable or invalid action.} \end{cases}$R(x,y)=⎩⎨⎧​1.0,−0.5,−1.0,​fa​=1,0≤fa​<1,unparsable or invalid action.​

The intermediate penalty distinguishes partially valid but incorrect actions from completely invalid outputs, preserving a meaningful reward gap for advantage estimation.

The final training objective combines clipped policy optimization with the platform‑conditioned MOPD penalty. For each sample $i$i, the token‑level advantage is computed from the structured reward using a group‑relative baseline:

$A_t^{(i)} = R(x^{(i)}, y^{(i)}) - \frac{1}{|g(i)|} \sum_{k \in g(i)} R(x^{(k)}, y^{(k)}),$At(i)​=R(x(i),y(i))−∣g(i)∣1​∑k∈g(i)​R(x(k),y(k)),

where $g(i)$g(i) is the prompt group. Let $r_t(\theta) = \frac{\pi_\theta(y_t \mid h_t)}{\pi_{\theta_{\mathrm{old}}}(y_t \mid h_t)}$rt​(θ)=πθold​​(yt​∣ht​)πθ​(yt​∣ht​)​ be the policy ratio. The per‑token clipped surrogate loss is

$\ell_{\mathrm{PG}}^{(t)}(\theta) = \min\left(r_t(\theta) A_t,\; \operatorname{clip}(r_t(\theta), 1-\epsilon_{\text{low}}, 1+\epsilon_{\text{high}}) A_t\right).$ℓPG(t)​(θ)=min(rt​(θ)At​,clip(rt​(θ),1−ϵlow​,1+ϵhigh​)At​).

The overall objective maximized for a platform‑$p$p rollout is

$\mathcal{J}(\theta) = \mathbb{E}_{p, x, y \sim \pi_\theta} \left[ \sum_{t} m_t \left( \ell_{\mathrm{PG}}^{(t)}(\theta) - \beta \mu \hat{D}_{\mathrm{KL}}^{(t,p)} \right) \right],$J(θ)=Ep,x,y∼πθ​​[∑t​mt​(ℓPG(t)​(θ)−βμD^KL(t,p)​)],

where $\beta$β controls the distillation strength. Equivalently, the loss minimized is

$\mathcal{L}(\theta) = \mathcal{L}_{\mathrm{PG}}(\theta) + \beta \mathcal{L}_{\mathrm{MOPD}}(\theta).$L(θ)=LPG​(θ)+βLMOPD​(θ).

This joint objective drives the policy toward higher task success through online RL while the routed teacher penalty anchors platform‑specific interaction patterns, preventing the shared policy from collapsing into a generic average behavior.

Experiment

The evaluation setup includes interactive desktop tasks on OSWorld, mobile tasks on MobileWorld, and static GUI grounding benchmarks to assess a single student agent trained with platform-conditioned distillation. Comparisons with mixed supervised fine-tuning and static model merging reveal that directly mixing heterogeneous supervision fails to improve both platforms consistently. UI-MOPD achieves balanced cross-platform performance by routing platform-specific teacher signals during online policy optimization, transferring expertise without sacrificing general GUI grounding ability.

UI-MOPD achieves the best balanced cross-platform performance, reaching 38.2% on OSWorld and 12.0% on MobileWorld. General vision-language models and specialized GUI agents show significant trade-offs between the two platforms, while integration strategies like mixed fine-tuning and static model merging fail to consistently improve both. UI-MOPD reaches 38.2% on OSWorld and 12.0% on MobileWorld, offering the best cross-platform balance among all compared methods. Single-platform GUI agents often sacrifice one domain for the other: GELab-Zero-4B scores 10.9% on MobileWorld but 31.9% on OSWorld, whereas GUI-Owl-7B reaches 34.9% on OSWorld but drops to 4.5% on MobileWorld.

Platform-specific fine-tuning of an 8B student on a single environment creates unbalanced transfer, improving one platform's performance while degrading the other. UI-MOPD trains the same student with platform-conditioned distillation from desktop and mobile teachers, yielding simultaneous gains of 4.3 points on OSWorld and 4.3 points on MobileWorld over the original 8B model. The resulting 8B student even surpasses the 32B base model on MobileWorld, showing that cross-platform transfer is not simply a matter of model scale. Fine-tuning the 8B model on OSWorld alone raises desktop success to 35.8% but zeroes out MobileWorld performance, while mobile-only fine-tuning reaches 12.8% on MobileWorld but does not match UI-MOPD's desktop improvement. UI-MOPD improves the 8B student on both platforms simultaneously, achieving 38.2% on OSWorld and 12.0% on MobileWorld, outperforming the 32B base model on MobileWorld despite using a smaller student. Platform-conditioned behavioral anchors in UI-MOPD prevent heterogeneous interaction signals from collapsing into a single averaged constraint, preserving platform-specific interaction patterns while boosting task success.

UI-MOPD improves mobile GUI task success on AndroidControl over the base Qwen3-VL-8B-Thinking model, while static parameter merging causes a notable drop. On grounding benchmarks, UI-MOPD closely preserves the base model's performance, whereas model merging leads to clear declines across all three datasets, indicating that the proposed optimization is less destructive to static GUI understanding. UI-MOPD achieves the highest AndroidControl accuracy (80.05%), outperforming both the base model and the TIES-merging checkpoint. Model merging reduces grounding performance substantially (e.g., ScreenSpot-Pro drops from 43.71% to 37.13%), while UI-MOPD maintains near-base scores across all grounding datasets.

The evaluation across OSWorld, MobileWorld, AndroidControl, and grounding benchmarks compares UI-MOPD’s platform-conditioned distillation against single-platform fine-tuning, model merging, and various GUI agents. UI-MOPD simultaneously improves an 8B student on both desktop and mobile tasks while preserving grounding accuracy, whereas alternative strategies create trade-offs or degrade performance. The results confirm that platform-conditioned behavioral anchors prevent heterogeneous interaction signals from collapsing, enabling effective cross-platform transfer without sacrificing static understanding.