LMSYS Arena published a coding-agents harness study comparing Claude Code, Codex CLI, and Pi. Across 21 rigorously tested model-harness pairs, the harness tax mattered far less than many assume—model quality still dominated. Details are on the Arena blog.
AI Research
AI Research
Specific Labs (@janaksunil) launched Real-SWE, evaluating coding agents on private, out-of-distribution production codebases from real companies (200K+ user apps, fintech statement pipelines, enterprise sales tools, and more). On the public board (8-run averages) Claude Fable 5.1 + Claude Code leads at 38.8%, GPT-6 Astra + Codex CLI at 33.8%, and Gemini 3.8 Flash + Gemini CLI at 31.2% — none clear 40%. Site: realswe.withspecific.com.
LMArena reports Muse Spark 1.3 (Max) at #13 on Agent Arena with +4.2% net improvement across 8.7K+ real agentic sessions — a 17-rank climb from Muse Spark 1.2 (xHigh) at #30. Code, Work, and Chat all improved, with Code moving to #15 (+4.3%).
OpenAI researcher Kevin Liu published Research acceleration metrics: coding agents now reshape daily researcher work; median researcher spends >$600/day of inference at API prices, research-org P90 >$7,000/day; as of mid-August 2026 the org uses ~3.1 agent-workdays per human workday. OpenAI says it has met its automated research-intern goal and is progressing toward an automated AI researcher around March 2028.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.