Evolving Agent Interfaces: Empowering Computer-Use Agents to Synthesize Dynamic APIs for 58.58% Success

Background and the Problem

The capability ceiling of modern Computer-Use Agents (CUAs) is fundamentally bounded by the human interfaces they are forced to navigate. Graphical user interfaces (GUIs) are tailored for biological vision and manual dexterity, fragmenting information across pop-ups, nested dropdowns, and scrolling canvases. Compelling an AI agent to screenshot, locate pixel coordinates, and click spreadsheet cells one by one incurs ballooning 50+ step action horizons where a single transient glitch collapses the entire workflow. Although foundation models excel at writing programmatic scripts, existing agent harnesses rigidly constrain them to static, human-designed tool sets. Researchers resolved this architectural mismatch in arXiv:2610.28601, introducing Evolving Agent Interfaces.

Architecture and How It Works

1. Autonomous Meta-Agent Interface Synthesis (Agent S4):

• Deploys a meta-agent equipped with an ephemeral scratch virtual machine mirrored to the target task environment;
• The meta-agent inspects target application internals, dynamically authoring, executing, and refining custom Python SDKs, CLI tools, and data-processing routines;

1. Persistent Repository Evolution:

• High-leverage interfaces that pass validation tests are committed to a structured interface repository;
• When downstream execution agents tackle tasks, they bypass fragile GUI click-streams, executing synthesized programmatic APIs directly;

1. Binary Success-Driven Selection:

• Selects evolving interface candidates based on hard end-to-end binary task completion, eliminating overfitting to superficial intermediate rewards.

Benchmarks and Measured Results

Evaluated on the premier OS-level agent evaluation suites, OSWorld V2 and ScienceBoard (Tables 2 & 3):

• Full Benchmark Task Resolution Surge:
• Across comprehensive evaluation suites, an open-weights 27B model augmented with evolving interfaces raised overall success from 55.03% to 58.58%;
• Solved 99 of 169 complex enterprise workflows compared to 93 for the static seed repository;
• Frontier-Class Efficiency:
• Establishes the highest binary success rate achieved by an open-weights model, rivaling commercial frontier models at a fraction of standard API inference costs;
• Compresses total action sequence lengths by over 40%, preventing the stochastic error compounding that plagues long-horizon GUI rollouts.

Getting Started for Developers

• Technical Report: Accessible on arXiv:2610.28601;
• Engineering Principles for Enterprise RPA: Developers designing desktop automation and robotic process automation (RPA) agents should transcend brittle pixel-matching paradigms. Equipping agents with scratch environments to synthesize bespoke programmatic wrappers for target software turns complex GUI tasks into deterministic script execution.