Teams from UC Berkeley, Princeton and others released FrontierCS 2 (arXiv 2610.29932): 48 tasks across 13 CS domains, each built by deleting a paper's contribution from the authors' released code. An agent has 5 hours to write it back and passes only if it beats the original on every metric and every hidden workload. Of 12 agents, Claude Opus 5.5 in Claude Code leads with a 58% pass rate, every other agent passes fewer than half, and 16 tasks have no passing run. On Oct 26 maintainer Wenhao Chai said a trace audit showed the Pi harness ends runs on refusals; rerun in Antigravity, Gemini 4 Argon now beats GPT-6 Astra and sits right behind Opus 5.5.
Key Takeaways
- ✓48 tasks, 13 domains, 12 agents, one run per agent per task (paper)
- ✓Strict pass rule: the final submission must beat the authors' code on every metric and every hidden workload; a higher total score can still fail
- ✓Claude Opus 5.5 in Claude Code leads with a 58% pass rate; 16 tasks have no passing run
- ✓Under the Pi harness, 10 of Gemini 4 Argon's 48 runs ended on provider refusals; rerun in Antigravity it beats GPT-6 Astra (Wenhao Chai's post)
- ✓Tasks and evaluators are open source on GitHub; the authors estimate a full 48-task run with Opus 5.5 at about $1,000 in API fees, $300 GPU and $100 CPU
Key Decision Metrics at a Glance
Turn your technical choice into a development budget
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Background and pain point
Benchmarks like SWE-bench and Terminal-Bench grade bug fixes against known tests. FrontierCS 2 asks something else: can an agent take over a paper's code and make its core method better than the authors did? It uses the authors' own released code as the bar, not a rubric or someone else's reimplementation.
What changed under the hood
Each task deletes the code for a paper's contribution from the paper's own repository, and the deleted code becomes the human reference. The agent gets the rest of the repo and may submit patches at any time within 5 hours; the judge returns only a development-suite score and one message. The evaluator, the hidden final suite and the reference stay in the judge's sandbox, and the agent's sandbox has no network except the driver, so it cannot fetch the authors' code. The agent picks its own final submission, and only that one is scored on the hidden workloads. The 48 tasks span 13 computer science domains, and a private set of more than 100 tasks will be announced later.
Benchmarks
The paper evaluates 12 models, each in one harness. Claude Opus 5.5 in Claude Code leads with a 58% pass rate, every other agent passes fewer than half, and no run passes 16 of the tasks. It also introduces the FCI capability index, a fork of Epoch AI's ECI anchored at Kimi K2.7 = 0; FCI and pass rate rank agents almost the same, and the median task sits at about the strongest agent's level. The site notes that 10 of Gemini 4 Argon's 48 runs under the Pi harness ended on provider refusals. On Oct 26 maintainer Wenhao Chai (Princeton PhD student, formerly at Google DeepMind) posted on X that rerun in Antigravity, Argon beats GPT-6 Astra and sits right behind Opus 5.5, a reminder that harness choice moves rankings as much as the model.
How to try it
The 48 tasks, evaluators and judge are open source on GitHub and install into your own AWS account; GPU tasks also need Modal, and any OpenAI-compatible API works for the model. The authors estimate one full run with Opus 5.5 at about $1,000 in API fees, $300 GPU and $100 CPU. When reading agent leaderboards, check retry, refusal handling and truncation policies too.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.