FocusGUI: Focused Omni-Modal GUI Agents Slashing Sequential Distraction and Doubling Type Match to 64.2%

Background and the Problem

As frontier AI transitions toward omni-modal streaming, modern graphical user interface (GUI) agents ingest continuous interleaved streams of high-resolution screen frames, synchronous ambient audio, and device state histories. However, during live mobile and desktop operating tasks, information relevance is exceptionally transient and sparse: a temporary keyboard pop-up or notification chime crucial to step $t$ becomes obstructive noise at step $t+1$. Existing frameworks naively expose the entire multi-modal history uniformly to the backbone. This induces severe 'Sequential Distraction', where accumulated historical clutter dilutes model attention, triggering cascading misclicks and action hallucinations. In arXiv:2610.26203, Alibaba researchers and academic collaborators introduced FocusGUI to solve this bottleneck.

Architecture and How It Works

1. Non-Invasive Modular Focus Adapter:

• Rather than crudely cropping raw screenshots (which degrades global spatial coordinates), FocusGUI freezes the underlying foundation backbone and inserts a lightweight modular attention filter;

1. Action-Sufficient Projection & Channel Decomposition:

• Projects heterogeneous evidence streams (UI pixels, audio wavelets, execution logs) into a shared 128-dimensional latent manifold;
• Dynamically weights multi-modal channels via task queries with calibrated softmax temperature ($ au=1.0$), compressing the observation into a dense 64-dimensional action-sufficient token that eliminates transient clutter;

1. Step-Adaptive Temporal Prioritization:

• Adjusts decay rates for aged multi-modal signals, ensuring the decoder prioritizes actionable UI elements on the active viewport.

Benchmarks and Measured Results

Evaluated on the standardized OmniGUI benchmark spanning 29 popular smartphone applications, 708 real-world episodes, and 2,572 interactive execution steps (Table 1, Figure 10):

• Action Accuracy Breakthroughs:
• Across four prominent open-source omni-modal backbones (Qwen2.5-Omni, VITA-1.5, MiniCPM-o 4.5, Baichuan-Omni-1.5), FocusGUI doubled average Type Match from 31.0% to 64.2%;
• Exact Match surged from 11.2% to 33.5% (nearly a 3x leap);
• Reached 1.9x the Type Match of the strongest competitive GUI enhancement baselines;
• Systematic Error Rectification:
• Across all 5 application domains, corrective action adjustments outnumbered regressions by over 150 instances, validating consistent resilience against cascading trajectory drift.

Getting Started for Developers

• Technical Report: Accessible via arXiv:2610.26203;
• Engineering Recommendations: Developers designing mobile copilots, smart automotive cockpit assistants, or desktop automation agents should avoid dumping raw streaming video and audio buffers into context windows. Adopting FocusGUI's channel decomposition provides immense performance gains without the compute overhead of full-model retraining.