Benchmark12d ago
FrontierCS 2 benchmark: agents must rebuild a paper's deleted contribution in its own repo and beat the authors' code; Claude Opus 5.5 leads at 58%, 16 tasks unsolved
Teams from UC Berkeley, Princeton and others released FrontierCS 2 (arXiv 2610.29932): 48 tasks across 13 CS domains, each built by deleting a paper's contribution from the authors' released code. An agent has 5 hours to write it back and passes only if it beats the original on every metric and every hidden workload. Of 12 agents, Claude Opus 5.5 in Claude Code leads with a 58% pass rate, every other agent passes fewer than half, and 16 tasks have no passing run. On Oct 26 maintainer Wenhao Chai said a trace audit showed the Pi harness ends runs on refusals; rerun in Antigravity, Gemini 4 Argon now beats GPT-6 Astra and sits right behind Opus 5.5.
Benchmark12d ago
Artificial Analysis Benchmarks DeepSeek Search API: $0.75/1K Searches Delivers +36 Point Score Lift
Independent AI evaluation platform Artificial Analysis (@ArtificialAnlys) officially benchmarked DeepSeek's web search API (DeepSeek Search). Augmenting the base model with DeepSeek Search boosted its Search Index score from 33 to 69—an impressive +36 point leap (ranking 17th among 27 tested search APIs). DeepSeek Search established a market-leading price-performance ratio, priced at just $0.75 per 1,000 searches (~$0.032 per completed task), making it the most cost-effective paid search API evaluated. It recorded 70% accuracy on AA-Omniscience and 63% on BrowseComp, highlighting an optimal balance between factual grounding and operational economics.
Research14d ago
BlenderWorld Benchmark Launches for Autonomous 3D Creation: GPT-6 Astra Leads with 79.9 Score on 223 Industrial Tasks
Researchers introduced BlenderWorld (arXiv:2610.23986), the first comprehensive benchmark evaluating visual agents creating and editing 3D models from real-world references in Blender. Spanning 223 tasks across procedural geometry, Python scripting, material shaders, spatial layouts, and lighting, GPT-6 Astra leads the benchmark with a composite Score of 79.9 (and a 42.2% Perfect completion rate), followed by Opus-5 at 73.0. Open-weight flagship DeepSeek-V4.1-Flash claims 65.7, matching proprietary models while relying on active execution-and-inspection tool loops.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.