AI Coding Live News

Vercel Labs launches agent-desktop: computer use for any coding agent via OS accessibility trees, not pixels, on macOS, Windows and Linux

On October 31, Chris Tate of Vercel Labs, creator of agent-browser, announced agent-desktop on X as "agent-browser, for your whole computer". It is an Apache-2.0 Rust CLI and C library that reads a desktop app's OS accessibility tree and returns every actionable element as a JSON ref; the agent clicks or types on a ref and snapshots again to verify, with no pixel matching, headless by default. Version 1.0.0 shipped on October 27 and added Linux support; the current npm release is 1.0.2.

Vercel Labs launches agent-desktop: computer use for any coding agent via OS accessibility trees, not pixels, on macOS, Windows and Linux

Key Takeaways

  • Install: npm install -g agent-desktop, one package with native x64/ARM64 binaries; 1.0.0 (Oct 27) added Linux (glibc 2.35+, via AT-SPI), and the latest npm release is 1.0.2 (Oct 30)
  • Loop: snapshot prints a JSON tree with refs, click @<snapshot_id>:eN acts, then snapshot again to verify; a changed UI returns STALE_REF, and errors say whether a retry is safe
  • Scope: the README lists 60 command names, 56 of them operational, spanning windows, keyboard and mouse, notifications, clipboard, waits and session traces
  • Headless by default: ref actions go through accessibility APIs without moving the pointer or stealing focus, with --headed for physical input; Chris Tate says multiple agents can use the computer in parallel, each with its own cursor
  • Benchmarks: no official success-rate or speed numbers; it is a tool with no model inside, and web pages stay with agent-browser
Read full story

X discussion: Box CEO Aaron Levie says under 1% of enterprise workflows really use AI agents, echoing Sarah Guo's "early Claude Code era" take

Conviction founder Sarah Guo posted on Oct 27 that enterprise workflow automation is "in the autocomplete / early claude code era", "but not for long", drawing about 80K views and nearly 500 likes. Box CEO Aaron Levie quoted it the next day and argued that most enterprise workflows have not had agents applied in any meaningful way, putting deployment at probably under 1%, mainly because of AI capability itself; he says computer use, long-running agents and agent swarms are changing that. His post has about 635 likes, 486 bookmarks, 63K views and 150+ replies. These are opinions and rough estimates, not measured statistics.

Key Takeaways

  • Sarah Guo (Conviction founder): enterprise workflow automation is still in the "autocomplete / early claude code era", but not for long
  • Aaron Levie (Box CEO): most enterprise workflows have not had agents applied meaningfully, and deployment across the economy is "probably < 1%", which is his estimate
  • Levie says the main bottleneck is AI capability itself, with enterprise adoption speed a smaller factor
  • He points to computer use, long-running agents, harness improvements and agent swarms as what now makes more complex workflows workable for agents
  • His view: the shift is less about users prompting agents and more about organizations re-engineering workflows, with the biggest impact in finance, life sciences, healthcare and other information-heavy fields
Read full story

OpenAI launches GPT-6.1 Terra: fast, low-cost computer use at $1/$5 per million tokens, half the price of GPT-5.6 Terra and GPT-6.1 Sol

On Oct 26 OpenAI released GPT-6.1 Terra, an efficient model for visual reasoning, precise computer use and natural writing. It is rolling out today to Plus, Pro, Enterprise and Edu users in ChatGPT Work and Codex, and in the API as gpt-6.1-terra. Standard API pricing is $1 input, $0.05 cached input and $5 output per million tokens, which OpenAI says is 50% lower than GPT-5.6 Terra and GPT-6.1 Sol, with a 1,050,000-token context window. By OpenAI's figures it scores 15 points higher than GPT-5.6 Terra on Agents' Last Exam at medium effort for about a third of the cost, and solves 50.3% of Terminal-Bench 4.0 at max effort.

OpenAI launches GPT-6.1 Terra: fast, low-cost computer use at $1/$5 per million tokens, half the price of GPT-5.6 Terra and GPT-6.1 Sol

Key Takeaways

  • Pricing (official docs): $1 input, $0.05 cached input, $1.25 for 30-minute cache writes and $5 output per million tokens; requests over 272K input tokens are billed at 2x input and 1.5x output; Batch/Flex are half price and Fast mode is 2x
  • Specs: 1,050,000-token context, 922,000 max input and 128,000 max output tokens, Jun 1 2026 knowledge cutoff; reasoning.effort from none to max, and tool calling requires the Responses API
  • Computer use (OpenAI figures): on Agents' Last Exam at medium effort it beats GPT-5.6 Terra by 15 points at about one-third the cost, and matches GPT-6 Astra at one-tenth the price per token; it finishes tasks over 3x faster than GPT-6 Astra and GPT-6.1 Sol
  • Coding and factuality (OpenAI figures): 50.3% on Terminal-Bench 4.0 at max effort, more than double GPT-5.6 Terra's published 21.5%; error rate on an internal error-inducing prompt set fell from 53.6% to 16.7% at low effort
  • Safety: OpenAI says it meets the Critical cybersecurity threshold under its Preparedness Framework and keeps the safeguards built for GPT-6 Astra
Read full story

Cursor launches Computer Use: agents can control your computer and drive desktop apps like the iOS Simulator to test what they build

On 2026-10-23 (15:59 UTC) the official Cursor account @cursor_ai announced "Cursor can now control your computer": agents can drive desktop apps such as the iOS Simulator to test the features they build. A follow-up post says Computer Use lets agents click and type across any app on your computer, not just the browser. Cursor turns it on by itself when a task needs it, and you can also request it by @-mentioning Computer Use in your prompt; updating Cursor is enough to try it. A community changelog in September mentioned Computer Use on Mac in version 3.20.21, and this is the official launch. Supported platforms, the permission model and benchmarks were not published.

Cursor launches Computer Use: agents can control your computer and drive desktop apps like the iOS Simulator to test what they build

Key Takeaways

  • Cursor officially says its agents can now control your computer and drive desktop apps like the iOS Simulator to test the features they build
  • Computer Use lets agents click and type across any app, not just the browser
  • Cursor uses it automatically when a task needs it, and you can request it by @-mentioning Computer Use in a prompt
  • Update Cursor to try it; supported platforms, the permission model and benchmarks were not published
Read full story
ADSponsored

Alibaba Tongyi Labs Open-Sources OpenGCUA: Full-Stack Generalist Computer-Use Agent Cutting OSWorld Steps by 53%

Alibaba Tongyi Labs (Tongyi-MAI) has published and open-sourced OpenGCUA (arXiv:2610.22200), establishing a fully open software stack for Generalist Computer Use Agents. Moving beyond brittle single-click GUI paradigms, OpenGCUA introduces a 'Code-Execution-as-Action' framework combining visual screen perception, accessibility trees, bash scripts, and Playwright APIs into unified programmatic execution blocks. Evaluated across eight frontier models on OSWorld-v2, OpenGCUA boosted mean scores by +19.6 points while cutting task steps by 53.0%, releasing an Apache-2.0 macOS menu-bar harness and Cross-Device SDK.

Alibaba Tongyi Labs Open-Sources OpenGCUA: Full-Stack Generalist Computer-Use Agent Cutting OSWorld Steps by 53%

Key Takeaways

  • OpenGCUA published in arXiv:2610.22200 and released under Apache 2.0 as an open generalist computer-use agent stack
  • Pioneers 'Code-Execution-as-Action', synthesizing loops, conditionals, and programmatic APIs into unified Python/Bash blocks
  • On OSWorld-v2, boosts mean score by +19.6 points across eight frontier models while reducing steps per task by 53.0%
  • Releases a native macOS menu-bar harness for physical execution alongside a Cross-Device SDK spanning PC, web, and mobile
Read full story

Alibaba Tongyi-MAI Releases OpenGCUA and MAI-Gym: Full Open Stack for Generalist Computer Use Boosts OSWorld-v2 by +19.6 Points and Cuts Steps by 53%

Alibaba's Tongyi-MAI team has open-sourced two foundational frameworks for generalist computer-use agents: OpenGCUA (arXiv:2610.22200) and MAI-Gym (arXiv:2610.20939). By unifying GUI interaction with programmatic code execution and harness-to-model decision transfer, OpenGCUA elevates scores on OSWorld-v2 across all eight tested models—including GPT-6 Astra and Claude Opus 5—by +19.6 points while cutting steps by 53.0%. Meanwhile, MAI-Gym scales online RL to 20,000 parallel instances, training MAI-Gym-9B to 78.4% on MobileWorld and 76.2% on OSWorld-Verified.

Alibaba Tongyi-MAI Releases OpenGCUA and MAI-Gym: Full Open Stack for Generalist Computer Use Boosts OSWorld-v2 by +19.6 Points and Cuts Steps by 53%

Key Takeaways

  • OpenGCUA boosts OSWorld-v2 by +19.6 points: unifies GUI and code execution across 8 frontier models while cutting average steps per task by 53.0%
  • Uncovers 19.4x real-device token efficiency gap: real macOS harness testing shows Qwen3.8-Max uses 19.4x as many reasoning tokens as GPT-6 Astra
  • MAI-Gym scales GUI RL to 20,000 parallel instances: MAI-Gym-9B hits 78.4% on MobileWorld and 76.2% on OSWorld-Verified, transferring mocked-app RL to physical phones (28.8% -> 48.1%)
Read full story

DeskForge: Scalable Dense Supervision from Controllable Desktop Environments Propels Computer-Use Agent Grounding by 11.5%

Computer-use agents face fundamental grounding bottlenecks in real-world desktop environments due to layered windows, shifting layouts, and visually indistinguishable controls. Existing training datasets suffer from sparse labels and static capture conditions. Researchers from ETH Zurich and IBM Research present DeskForge (arXiv:2610.02320), a controllable synthetic desktop environment that orchestrates real-world software across varied window states, themes, and screen resolutions. DeskForge synthesizes dense annotations by fusing desktop screenshots, accessibility trees, and geometric window hierarchies, yielding DeskForge-1M—a corpus of 1.2M observations and 159.7M annotated UI element instances. Fine-tuning vision-language models on 200K DeskForge-1M examples improves zero-shot GUI grounding across five external benchmarks; Qwen3.5-4B gains 11.51 percentage points on ScreenSpot-Pro and 10.11 points on OSWorld-G. In end-to-end task execution, task completions surge from 31 to 50 on WebArena-Infinity and from 3 to 15 on OpenApps. The dataset, models, and code are open-sourced at saidgurbuz.github.io/deskforge.

Key Takeaways

  • ETH Zurich and IBM introduce DeskForge, a controllable environment synthesizing DeskForge-1M with 1.2M observations and 159.7M UI elements
  • Fine-tuning Qwen3.5-4B boosts GUI grounding by 11.51 points on ScreenSpot-Pro and 10.11 points on OSWorld-G across five external benchmarks
  • Multiplies long-horizon task completion: solves 50/119 tasks on WebArena-Infinity (up from 31) and 5x on OpenApps (from 3 to 15), fully open-sourced
Read full story

OSWorld-Pro: NVIDIA Unveils Process-Based Benchmark for Computer-Use Agents Spanning 2,800+ Subgoals

Computer-Use Agents (CUAs) designed to operate graphical desktop environments through virtual keyboard and mouse interactions are traditionally evaluated using outcome-based functional verifiers that assess only the final state after hundreds of operational turns. This black-box approach obscures whether failures stem from syntax typographical errors, GUI bounding-box misalignments, or intermediate state drift, preventing actionable diagnostic insights. NVIDIA Research introduces OSWorld-Pro (arXiv:2609.24890), the first comprehensive process-based evaluation suite for computer-use agents. Spanning 300+ complex desktop workflows decomposed into 2,800+ sequentially dependent subgoals grounded in 67,000+ expert human annotations, OSWorld-Pro deploys calibrated LLM judges to track incremental subgoal completion. Evaluations reveal that OSWorld-Pro presents a formidable challenge to state-of-the-art models: frontier models like Claude Opus 5 achieve only 75.7% (compared to 83.4% on standard OSWorld), systematically isolating critical procedural failure patterns such as exploratory irrelevant actions and click grounding drifts.

OSWorld-Pro: NVIDIA Unveils Process-Based Benchmark for Computer-Use Agents Spanning 2,800+ Subgoals

Key Takeaways

  • NVIDIA Research unveils OSWorld-Pro, decomposing 300+ desktop tasks into 2,800+ sequentially dependent subgoals
  • Grounded in 67,000+ expert annotations, revealing Claude Opus 5 scores only 75.7% vs 83.4% on standard OSWorld
  • Isolates subgoal-irrelevant actions and click coordinate misalignments, replacing black-box outcome verifiers with process diagnostics
Read full story
A
AICoder Editorial@aicoder·
👀77
🔥90

Developer 'A vs B' comparison intent surges: Sonnet 5.5, Sol, Argon, Air, and Copilot Computer Use collide in one week

Over ~7 days, Claude Sonnet 5.5, GPT-6.1 Sol, Gemini 4 Argon, JetBrains Air in IDEs EAP, and GitHub Copilot Computer Use landed in the same window. Search and HN discourse shifted from single-release chase to pairwise selection on price bands, terminal agents, IDE harnesses, and computer-use. Meta industry brief for readers without X.

Key Takeaways

  • Same-week cascade: Sonnet 5.5 in Copilot (09-28) → Sol in Copilot (09-29) → Argon Fairwind preview (09-30) → Air EAP + Copilot Computer Use (10-01)
  • Hot pairs: Sonnet↔Opus, Sonnet↔Sol, Argon↔Opus/Astra, Air↔Cursor, Copilot CU↔Claude CU, Codex↔Claude Code, Sol↔Astra same-family economics
  • Selection axes: price band × terminal harness × IDE orchestration (Air ACP BYO) × computer-use for GUI-only legacy
  • Compliance caveat: Anthropic subscription auth ToS constrains third-party harnesses; Air docs allow Claude subscription via terminal full-screen tab only
  • Argon still Fairwind-gated (not GA); no public developer GA date—label comparison pages as preview-gated
Read full story

GitHub Copilot Computer Use enters public preview in CLI and Copilot app (macOS/Windows)

On 2026-10-01 GitHub announced Computer Use in public preview for Copilot CLI and the Copilot app on macOS and Windows: agents can drive local desktop apps via accessibility tree/screenshots, clicks, typing, scroll/drag, and cross-app workflows—targeting GUI-only legacy software. Off by default; enable with /computer on or Settings; enterprise policy can block it.

GitHub Copilot Computer Use enters public preview in CLI and Copilot app (macOS/Windows)

Key Takeaways

Read full story
ADSponsored

OpenAI Node 7.25.0: Computer Use on beta.agents; 7.24 adds GPT-6.1 Sol + Agents credential/session options

Official TypeScript SDK [email protected] (v7.25.0, 2026-09-29) adds Computer Use to beta.agents (#2837), mirroring Python 3.22.0. Prior v7.24.0 shipped Agents credential/session options (#2812), the gpt-6.1-sol model id (#2836), Responses Cyber access, Realtime Translation client secrets, and a dedicated Node translation WebSocket. 35 commits ahead of v7.23.0.

OpenAI Node 7.25.0: Computer Use on beta.agents; 7.24 adds GPT-6.1 Sol + Agents credential/session options

Key Takeaways

  • Releases v7.24.0 → v7.25.0; npm [email protected]; 35 commits ahead of v7.23.0
  • Computer Use on beta.agents via #2837
  • gpt-6.1-sol id (#2836); Agents credential/session options (#2812)
  • Realtime Translation client secret (#2819) + Node translation WebSocket (#2818)
  • Upgrade: npm i [email protected]; docs: platform.openai.com agents + API reference
Read full story

OpenAI Python 3.22.0: Computer Use on beta.agents; browser auth/screenshots + desktop environment templates

OpenAI’s Python SDK shipped v3.22.0 on 2026-09-29 (PyPI openai==3.22.0): typed computer-use tools on beta.agents, optional screenshots, browser authentication and origin-access approval inputs, plus desktop settings / environment templates for hosted environments. Node SDK v7.25.0 landed the same API surface the same day.

OpenAI Python 3.22.0: Computer Use on beta.agents; browser auth/screenshots + desktop environment templates

Key Takeaways

  • Release v3.22.0; PR #3989
  • beta.agents gains computer_use tools, screenshots, browser auth + origin-access approvals
  • Desktop settings / environment templates for hosted agent environments
  • Node twin: v7.25.0
  • Upgrade: pip install -U openai==3.22.0; guide: Agents
Read full story

Cursor Self-Hosted: one `agent worker start` turns a Mac Mini into a remote worker; `--computer-use` drives the desktop

Cursor engineer @mattyp shows a Mac Mini becoming a Cloud Agent worker via agent worker start; --computer-use enables remote click/type/screenshot through Cursor Computer Use. Official Self-Hosted Machines keep inference in Cursor cloud and tool execution on your hardware over outbound HTTPS (My Machines + Team Pools).

Cursor Self-Hosted: one `agent worker start` turns a Mac Mini into a remote worker; `--computer-use` drives the desktop

Key Takeaways

  • Product path: Self-Hosted Machines blog + docs — tools on your hardware, loop in Cursor cloud
  • Personal quickstart (My Machines): install CLI → agent login → agent worker start (outbound HTTPS only)
  • Desktop control: agent worker --computer-use start; grant Accessibility + Screen Recording to Cursor Computer Use on macOS
  • Scale: 200 workers/user, 1000/team; Team Pools need Enterprise + service-account API key
  • Verify with agent worker debug + screenshot task; route via worker= from Slack/GitHub/Linear
Read full story

Anthropic ships Claude Opus 5.5: Fable 5.1-level on most tasks, ~40% cheaper than Opus 5

Anthropic / Claude (@claudeai; confirmed by @AnthropicAI) launched Claude Opus 5.5 today, the first model in the new Claude 5.5 family. Officially it matches Claude Fable 5.1 on most tasks, and is a major step up from Opus 5 on agentic coding, computer use, and knowledge work. At default settings typical workloads cost about 40% less than Opus 5. Pro, Max, and Team five-hour usage limits are increased, with a bankable rate-limit reset for subscribers.

Anthropic ships Claude Opus 5.5: Fable 5.1-level on most tasks, ~40% cheaper than Opus 5

Key Takeaways

  • Opus 5.5 opens the Claude 5.5 family and is available today; most tasks match Claude Fable 5.1.
  • Leads Opus 5 on agentic coding, computer use, and knowledge work; ~40% cheaper on typical default workloads.
  • Higher Pro/Max/Team five-hour usage caps plus a bankable rate-limit reset for subscribers.
Read full story

Anthropic Releases Computer Use, Skills, and Files APIs to GA with New Browser Use Tool

Anthropic officially announced the General Availability (GA) of three core agent APIs: Computer Use (direct OS and application interaction), Skills API (dynamic workflow knowledge injection), and Files API. Concurrently, it launched Browser Use, enabling agents to navigate and manipulate web browsers autonomously.

Key Takeaways

  • Promotes Computer Use and Skills APIs to enterprise-grade GA status with full SLAs
  • Launches native Browser Use tool bridging terminal coding with live browser execution
  • Empowers enterprise developers to build end-to-end multi-app autonomous agents
Read full story