On October 31, Chris Tate of Vercel Labs, creator of agent-browser, announced agent-desktop on X as "agent-browser, for your whole computer". It is an Apache-2.0 Rust CLI and C library that reads a desktop app's OS accessibility tree and returns every actionable element as a JSON ref; the agent clicks or types on a ref and snapshots again to verify, with no pixel matching, headless by default. Version 1.0.0 shipped on October 27 and added Linux support; the current npm release is 1.0.2.
Key Takeaways
- ✓Install:
npm install -g agent-desktop, one package with native x64/ARM64 binaries; 1.0.0 (Oct 27) added Linux (glibc 2.35+, via AT-SPI), and the latest npm release is 1.0.2 (Oct 30) - ✓Loop:
snapshotprints a JSON tree with refs,click @<snapshot_id>:eNacts, thensnapshotagain to verify; a changed UI returnsSTALE_REF, and errors say whether a retry is safe - ✓Scope: the README lists 60 command names, 56 of them operational, spanning windows, keyboard and mouse, notifications, clipboard, waits and session traces
- ✓Headless by default: ref actions go through accessibility APIs without moving the pointer or stealing focus, with
--headedfor physical input; Chris Tate says multiple agents can use the computer in parallel, each with its own cursor - ✓Benchmarks: no official success-rate or speed numbers; it is a tool with no model inside, and web pages stay with agent-browser
Key Decision Metrics at a Glance
Turn your technical choice into a development budget
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Core Background & Industry Pain Points
Most desktop computer use today means screenshots plus a vision model guessing coordinates: slow, token-hungry, brittle when the UI shifts, and it takes over the user's mouse and focus. Vercel Labs' agent-browser solved this for web pages with accessibility snapshots, but agents testing Electron apps or driving Finder or system settings fell back to pixels. agent-desktop brings the same approach to the whole desktop.
Architecture Highlights & Internals
Per the GitHub README, agent-desktop is written in Rust and ships as a CLI and a C library (libagent_desktop_ffi). It reads the OS accessibility tree (Accessibility on macOS, UI Automation on Windows, AT-SPI on Linux) and numbers each actionable element as @<snapshot_id>:eN. Refs belong to one snapshot, so after a UI change a stale ref returns STALE_REF instead of hitting a look-alike element. Every command prints one JSON envelope, and errors state whether a retry is safe. It runs headless by default, without moving the pointer or taking focus; --headed allows physical input. There is no model inside: the observe, act and verify loop stays in the calling agent.
Authoritative Benchmarks & Measured Scores
No official success-rate, latency or head-to-head numbers against screenshot-based computer use have been published. The verifiable figures are engineering ones: the README lists 60 command names, 56 operational; 1.0.0 shipped on October 27, and npm shows 46 versions with 1.0.2 the latest. The claim that several agents can work in parallel, each with its own cursor, comes from Chris Tate's post and has not been independently tested.
Developer Hands-on Guide
Run npm install -g agent-desktop, then agent-desktop status; on macOS grant Accessibility access via permissions --guide. Point AGENTS.md at agent-desktop skills get desktop, or install the skill with npx skills add, so Claude Code, Codex and others can call it. For dense apps, map with --skeleton first and drill down. Snapshots, screenshots and clipboard reads can contain private data, so keep them out of public logs.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.