Existing Computer-Use Agents (CUAs) face a dilemma: relying strictly on graphical user interface (GUI) interactions is slow and error-prone, while building per-application API tools demands unsustainable engineering and lacks generalizability. Researchers from Zhejiang University introduce HybridCUA, combining the UI versatility of GUIs with the execution speed and determinism of the Command Line Interface (CLI). Supported by the newly constructed HybridCUA-8K dataset (5K hybrid trajectories and 3K RLVR tasks), HybridCUA employs a two-stage training paradigm: supervised fine-tuning across interleaved GUI-CLI modalities, followed by CLI-aware reinforcement learning that rewards reliable, selective shell execution. On the OSWorld benchmark, HybridCUA-9B achieves 53.6% accuracy, improving over the base model by 14.8 percentage points, while delivering a 4.0-point gain on WindowsAgentArena.

Key Takeaways

  • ✓Unified GUI and CLI Paradigm: Bridges the gap between fragile vision-only screen clicking and application-specific APIs, combining the speed of shell commands with the visual ubiquity of desktop GUIs.
  • ✓HybridCUA-8K Trajectory Dataset: Curates 5,000 interleaved GUI-CLI trajectories and 3,000 verified RLVR tasks with ground-truth environment state feedback for agent post-training.
  • ✓53.6% on OSWorld with Cross-OS Generalization: HybridCUA-9B hits 53.6% accuracy on OSWorld (a 14.8 percentage point gain over base) and boosts WindowsAgentArena by 4.0 points across distinct OS environments.
🧭

Finished reading? Explore benchmark rankings & pricing

Real-world SWE-bench scores & $20/mo vs API cost break-even calculator

🏆 Leaderboard→💰 Compare Plans↗
🔗

Project Links & Resources

Direct Access
Direct access to official project resources and documentation
🐙
GitHub Repository (ZJU-REAL/HybridCUA)
github.com/ZJU-REAL/HybridCUA
↗
🌐
ArXiv Preprint (2609.38008)
arxiv.org/abs/2609.38008
↗
📖
Hugging Face Daily Papers (2609.38008)
huggingface.co/papers/2609.38008
↗
🔬

In-Depth Technical Analysis

Core Background & Industry Pain Points Computer-Use Agents (CUAs) designed to navigate operating systems autonomously have rapidly gained traction. However, prevailing implementations face severe architectural limitations: 1. Fragility and Inefficiency of Vision-Only GUIs: Relying purely on visual screenshots and simulated cursor clicks makes operations painfully slow. Navigating nested directories, bulk text replacements, or parsing logs via UI requires dozens of high-latency inference calls, easily derailed by transient UI pops or rendering glitches; 2. API Tool Bloat and Poor Generalizability: Providing bespoke API tools for every individual desktop application requires immense engineering labor and fails to scale across thousands of third-party software programs. The industry urgently requires an architecture that harmonizes the generality of GUIs with the deterministic efficiency of the Command Line Interface (CLI). ### Architectural Highlights & Underlying Mechanics Zhejiang University researchers introduce HybridCUA, a principled framework orchestrating multimodal GUI perception with CLI shell execution: 1. HybridCUA-8K Trajectory Suite: Curates 5,000 interleaved GUI-CLI execution traces alongside 3,000 verifiable reinforcement learning tasks (RLVR) with verified environment outcomes, spanning GUI-only, CLI-only, and interleaved trajectories; 2. Two-Stage Agent Post-Training: - Stage 1 (Interleaved SFT): Aligns multimodal representations across desktop screenshots, window trees, and terminal stdout/stdin streams; - Stage 2 (CLI-Aware RL): Employs outcome-driven reinforcement learning with reward signals penalizing inefficient visual clicking when concise shell commands suffice, while penalizing syntax or environment errors to cultivate reliable execution. ### Benchmark & Experimental Validation - 53.6% on OSWorld (+14.8% Leap): On the demanding Linux OSWorld benchmark, HybridCUA-9B achieves a 53.6% task completion rate, surging 14.8 percentage points over its base foundation model and setting a high-water mark for open-weights desktop agents; - Cross-Platform Verification on Windows: Without bespoke architectural tuning, HybridCUA delivers a +4.0 percentage point gain on WindowsAgentArena, confirming seamless generalization across differing OS kernels; - 58% Fewer Operational Steps: On repetitive administrative workflows (regex batch renaming, configuration patching), offloading operations to shell commands halved end-to-end latency and cut step counts by 58%. ### Engineering Takeaways & Practical Guide - Code & Dataset Access: The full codebase is available on GitHub at ZJU-REAL/HybridCUA, and the paper is indexed at arXiv:2609.38008; - Blueprint for Enterprise Desktop Agents: Production desktop agents should not rely solely on screenshot OCR and coordinate clicking. Implementing sandboxed shell execution alongside visual interfaces yields faster throughput and drastically lower token overhead; - Future Directions: Ongoing extensions target unified action spaces incorporating macOS AppleScript and mobile ADB pipes.