One-sentence Summary

Researchers from Nanyang Technological University and Alibaba Group propose FocusGUI, a lightweight adapter that dynamically prioritizes omni-modal GUI channels against step-specific decision references, compresses the most relevant evidence into a compact latent, and subtly injects it into frozen decoder embeddings to guide attention without altering the backbone's native parameter space; trained solely on action supervision, FocusGUI improves Type Match by $+33.2$+33.2 and Exact Match by $3.0\times$3.0× on average across four omni-modal backbones.

Key Contributions

• The paper formulates Focused Perception as a step-adaptive, action-sufficient, and non-intrusive representation requirement for omni-modal GUI agents, addressing the compounding errors of Sequential Distraction.
• FocusGUI is introduced as a lightweight adapter that dynamically prioritizes multimodal channels against a step-specific decision reference, compresses the most relevant evidence into a compact latent, and injects it into the frozen decoder through a zero-initialized residual at the action boundary.
• Trained from action supervision alone, without modality-relevance labels or backbone fine-tuning, FocusGUI improves over four omni-modal backbones by an average of +33.2 Type Match and 3.0x Exact Match over frozen backbones, and by +30.1 Type Match over the strongest existing enhancement method.

Introduction

GUI agents now operate on streaming omni-modal observations in which the evidence for the next action may come from audio, video, device state, or visual changes rather than only the current screenshot. This step-level relevance matters because closed-loop agents feed executed actions back into the next observation, so misreading one step can compound into trajectory drift. Prior work often responds by spatially cropping the screen, but that approach ignores non-visual signals and can remove the background context needed to ground coordinates and text; the deeper limitation is that the decoder lacks explicit guidance about what changed at the current step. The authors propose FocusGUI, which keeps a frozen omni-modal backbone and its full input unchanged, scores separable perceptual channels against a reference derived from the current observation, compresses the prioritized channels into a focus latent, and injects that latent through a zero-initialized residual at the action decoder boundary using only next-action supervision.

Dataset

The authors evaluate on OmniGUI, a benchmark for omni-modal smartphone GUI interactions.

• Source and composition: OmniGUI covers GUI interactions across 29 applications.
• Scale: The benchmark contains 708 episodes with 2,572 action steps.
• Observation schema: Each action step pairs a screenshot with synchronous audio, video, and action history. The decisive signal can come from any of these four streams.
• Ground truth and metrics:
  • Type Match (TM): checks whether the predicted action primitive matches ground truth.
  • Exact Match (EM): also requires predicted coordinates to fall within the target element’s bounding box and generated text to match the reference.
  • Goal Progress (GP): the fraction of correctly executed steps per episode, averaged across episodes.
• Split and usage: The authors use an episode-disjoint split, applied consistently across all four backbones and baselines. The frozen backbones are not updated; training updates only the lightweight FocusGUI head and action predictors.
• Processing details in FocusGUI: Evidence channels are projected into a shared 128-dimensional space, weighted by the full-modal query, compressed to a 64-dimensional action-sufficient embedding, and expanded into K = 4 residual tokens added to the last prompt embeddings before the action boundary.
• Filtering and cropping: The provided excerpt does not specify dataset filtering rules or cropping strategy.

Method

The authors formulate GUI interaction as a next-action prediction problem over omni-modal observations. At each interaction step $t$t, the agent receives an observation

where $g$g is the user goal, $I_t$It​ is the current interface, $V_t$Vt​ is temporal visual observation, $A_t$At​ is acoustic observation, $H_t$Ht​ is interaction history, and $S_t$St​ is device state. The model predicts an executable action sequence

which specifies both the action primitive and its arguments.

FocusGUI is built around a frozen pretrained omni-modal backbone with parameters $\Theta_0$Θ0​. Rather than pruning or replacing any part of the observation, the method keeps the original multimodal pathway intact and adds a lightweight auxiliary focus pathway. From the same observation, FocusGUI constructs a compact focus representation

maps it to a residual

and injects that residual into a small set of existing embedding positions. The decoder therefore operates on

where $E_t$Et​ is the original embedding sequence presented to the decoder, $J_t$Jt​ denotes the selected positions, and $\mathcal{S}_{J_t}$SJt​​ places the residual only at those positions. This design separates observation preservation from decision guidance: the original pathway retains all multimodal evidence, while the auxiliary pathway learns how the next action should be emphasized.

To make source priorities explicit, FocusGUI first exposes the observation as separable perceptual channels

where $\mathcal{M}$M indexes individual modalities together with the backbone hidden state and a spatial screen channel. Each channel is projected into a shared $d$d-dimensional space through a source-specific projector,

A decision reference $q_t$qt​ is derived from the integrated representation $e_t^{(\mathrm{all})}$et(all)​ via

which summarizes the joint observation context. For each channel, FocusGUI computes a compatibility score with this joint context and normalizes the scores into step-specific priorities:

Here $r_m$rm​ is a learned channel-identity embedding and $\tau$τ is the softmax temperature. Since the perceptual channels and joint context are recomputed at every step, the relative emphasis among information sources can change across the task without carrying state between steps.

The prioritized representation $c_t$ct​ is then compressed into a lower-dimensional focus vector

This projection constrains the capacity of the conditioning signal rather than the information accessible to the backbone. To make the focus usable by the frozen decoder, FocusGUI aligns it with the decoder embedding space at the action boundary. Let $b_t$bt​ denote the position where action generation begins, and let

be the $K$K preceding embedding positions. The compact focus vector is expanded into a residual over this window:

The residual is injected immediately before action generation, so the focus signal is available to all subsequent action tokens without requiring a separate prediction head or altering the output vocabulary. The additive form preserves the original embeddings and treats focus only as an additional conditioning signal. The parameters $W_p$Wp​ and $b_p$bp​ are initialized to zero, so training starts from the original frozen-model input and gradually learns observation-conditioned deviations. The method does not append textual hints or remove any multimodal input.

All FocusGUI components are learned jointly from action supervision only, with no annotations indicating which modality or channel should be emphasized. Let $\phi$ϕ denote the trainable parameters. The training objective is

The loss is computed only over action tokens. Gradients propagate through the frozen decoder to the auxiliary parameters $\phi$ϕ, while the omni-modal backbone, decoder, and language-model head remain unchanged. At inference time, FocusGUI constructs perceptual channels from the current observation, evaluates them under the current joint semantic context, forms the compact focus representation, and generates the corresponding residual. The frozen decoder then autoregressively produces the next executable action through its original generation pathway.

Experiment

The experiments evaluate FocusGUI on OmniGUI, an omni-modal smartphone GUI benchmark across 29 apps, using four frozen open-source backbone models and comparing it with screen-refinement baselines and a trained action head. The main results show that FocusGUI consistently improves action prediction across all backbones, while screen refinement fails because it discards surrounding context and cannot access non-visual channels. Ablation and interaction-depth analyses indicate that all components contribute most strongly to episode-level goal progress, that FocusGUI maintains accuracy as trajectories lengthen while the frozen backbones degrade, and that learned channel priorities shift from interaction state to audio and video across an episode. Capacity studies further show that the method remains stable across a wide parameter range without requiring per-backbone tuning.

Across four frozen omni-modal backbones, FocusGUI consistently improves task matching, exact match, and goal progress over direct inference and prior screen-refinement baselines. The gains span localization, semantic, cross-modal, temporal, and instant task categories, with the largest improvements in temporal settings and margins that widen as interaction depth increases. Screen-refinement baselines remain near neutral on task matching but reduce exact match and goal progress, indicating that narrowing a screenshot discards context needed for accurate grounding. FocusGUI achieves the best task matching and exact match across every backbone and task category tested. Screen-refinement baselines are near neutral on task matching but consistently lower exact match and goal progress, showing that refining screen regions cannot recover missing cross-modal or temporal context.

Ablating any of the four components lowers overall performance, and the losses are largest for episode-level Goal Progress. The components have a disproportionately stronger effect on Goal Progress than on single-step Type Match, indicating they mainly prevent trajectories from unraveling rather than only improving individual action selection. Channel decomposition and prioritization jointly improve stability, while residual alignment primarily supports grounding selected actions into correct arguments. Every removed component has a larger relative impact on Goal Progress than on Type Match, with up to about a third of Goal Progress lost compared with under a tenth of Type Match. Channel decomposition without step-adaptive prioritization raises Type Match and Exact Match over a single integrated channel, but prioritization is needed to convert richer channel access into episode-level stability. Residual alignment affects Goal Progress and Exact Match much more than Type Match, indicating it helps ground the chosen action primitive into correct arguments.

Performance is largely insensitive to the boundary window size across a wide sweep, with only small fluctuations in exact match, goal progress, and temporal goal progress and no consistent trend. The compact focus dimension has a clearer effect: reducing it to 32 substantially lowers exact match and goal progress, while the default setting remains stable and reports the highest exact match in the sweep. The results justify fixing both capacity parameters across backbones without per-benchmark tuning. Changing the boundary window across a sixteenfold range produces only small changes in overall metrics and no monotonic trend. Using a compact focus dimension of 32 instead of the default reduces exact match and goal progress sharply, indicating the latent representation becomes capacity-limited. The default parameter setting yields the highest exact match in the reported sweep and stable overall performance.

Across four frozen omni-modal backbones, FocusGUI consistently outperforms direct inference and screen-refinement baselines on task matching, exact match, and goal progress, with the largest gains in temporal tasks and deeper interactions, while screen refinement discards context and hurts exact grounding. Ablation shows all four components contribute mainly to episode-level stability rather than single-step action selection, with channel decomposition plus prioritization improving trajectory coherence and residual alignment supporting argument grounding. Performance is largely insensitive to boundary window size, whereas a too-small focus dimension sharply limits capacity and reduces exact match and goal progress, supporting the default settings without per-benchmark tuning.