Tooling8d ago
Microsoft Research and UC Berkeley open-source TokenEconomizer, a plugin that cuts Claude Code and Codex costs by 18.4–30.9% on SWE-Bench without changing the agent
Microsoft Research, Microsoft Security and UC Berkeley open-sourced TokenEconomizer on GitHub (Apache 2.0, v0.1.0 released October 26), a plugin for Claude Code, Codex, GitHub Copilot CLI and Pi, with experimental opencode support. It learns from earlier sessions in the same repository: micro-skills remember repo facts, tool pruning drops tool definitions that go unused, and a loop guard nudges an agent that is stuck. The paper reports 18.4–30.9% lower mean cost across five agent and model combinations on 277 SWE-Bench tasks (paper figures).
Benchmark14d ago
Artificial Analysis Benchmarks Devin Fusion: Scoring 68.1 on Coding Agent Index While Slashing Task Cost by 31%
Artificial Analysis released an independent evaluation of Cognition's new coding agent, Devin Fusion. Moving beyond single-model execution, Devin Fusion employs an asymmetric multi-agent architecture pairing Claude Opus 5.5 (max) as the lead reasoning orchestrator alongside GPT-6 Luna (high) as a high-speed parallel worker. In the standardized Coding Agent Index, Devin Fusion achieved an elite score of 68.1 (matching Muse Code at 68.0 and rivaling Claude Code Sonnet 5.5 at 68.4). Crucially, the hybrid architecture slashes average task expense from $13.04 down to $9.01 (a 31% cost reduction) while shortening execution runtime from 65 to 55 minutes.
Agents21d ago
ProgramGym Open-Sourced: Scaling 1,648 Verifiable Environments for Long-Horizon Program Reproduction Agents
Current software engineering benchmarks and training pipelines predominantly evaluate coding agents on localized bug fixing within existing repositories (such as SWE-bench) or isolated code snippet generation. They fail to teach agents how to explore unfamiliar reference systems, formulate architectural blueprints, and reproduce complete, multi-file software projects from scratch under long-horizon constraints. Researchers introduce ProgramGym (arXiv:2610.13395, repo: github.com/Acm1011/ProgramGym, dataset: ProgramGym/programgym-1648), an automated framework scaling verifiable training environments for program reproduction agents. ProgramGym consists of Program-Env, a feedback-driven engine that converts open-source repositories into verified runtime sandboxes paired with black-box test suites across C, C++, Go, and Rust (generating ProgramGym-1648), alongside Program-Agent, an iterative workflow enforcing role isolation across Analysis, Implementation, and Review stages with revision management. Fine-tuning Qwen3.8-27B on ProgramGym trajectories raises ProgramBench test pass rates from 34.87% to 42.84% (+7.97 points), outperforming GPT-5.4 under mini-SWE-agent (37.70%). Coupling SFT with the Program-Agent scaffold boosts performance to 56.25%, eclipsing Claude Opus 4.7 (50.90%) and Gemini 3.5 Flash (53.60%), with datasets and codebase open-sourced.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.