Cognition, the maker of Devin, released FrontierCode 2.0 on Oct 27. It keeps the method of maintainer-written tasks graded on whether the code is actually mergeable, but draws tasks from more repositories, adds security patches and performance work, uses deliberately underspecified prompts and adds many thoroughness criteria. In Cognition's results Claude Opus 5.5 leads at 45.5%, GPT-6 Astra follows at 42.8%, and Grok 4.7 and Gemini 4 Argon drop into the 20s, leaving the benchmark far from saturated.
Key Takeaways
- ✓Claude Opus 5.5 leads at 45.5% (54.6% on v1.1) and GPT-6 Astra scores 42.8% (53.3% on v1.1), per Cognition (vendor-reported)
- ✓Grok 4.7 scores 28.6%, Gemini 4 Argon 26.3%, Muse Wonder 1.0 24.3%, SWE-2 23.9%, DeepSeek V4.1 Flash 19.3% and Kimi K3 17.7%, showing a wide gap for open models
- ✓Versus v1.1, reference solutions touch 67% more files and 89% more lines in codebases 25% larger; tasks come from 55% more repositories with 43% more blocking criteria
- ✓Thoroughness blockers grew from 168 to 501, about 3x; in 80% of tasks the reference solution improves on the upstream PR, and an audit agent found and reported an upstream security vulnerability
- ✓On the cost-score Pareto frontier, Opus 5.5 sits at the high end and Luna, Haiku and Sol at the low-cost end, with GPT-6.1 Sol called out for strong value (official blog)
Key Decision Metrics at a Glance
Turn your technical choice into a development budget
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Background and industry pain points
Most coding benchmarks test whether a model can write code that passes tests against a detailed spec, and they saturate quickly. Deploying Devin, Cognition found that models still struggle to infer intent, respect unstated constraints and anticipate long-term consequences, which forces humans to step in. FrontierCode 1.0 reframed the question as "would you actually merge this PR?", and 2.0 asks "should you?"
How the benchmark works
Version 2.0 keeps the 1.0 method: more than 20 well-known open-source maintainers each spent over 40 hours per task building evals from repositories they maintain, with rubrics for mergeability, and any solution that fails a blocking criterion scores 0. The upgrades are broader coverage (55% more repositories, including security patches and performance work), 43% more blocking criteria with a new emphasis on thoroughness, such as fixing the same bug in a sibling code path, and larger task scope. Prompts are deliberately underspecified so models must use judgment. Cognition also loosened the controversial scope criteria, cutting the scope failure rate from 9.0% to 3.8%.
Reported results
All scores are Cognition's own, each model at its best reasoning effort: Claude Opus 5.5 45.5%, GPT-6 Astra 42.8%, Grok 4.7 28.6%, Gemini 4 Argon 26.3%, Muse Wonder 1.0 24.3%, SWE-2 23.9%, DeepSeek V4.1 Flash 19.3% and Kimi K3 17.7%. Scores fall versus v1.1 and tiers separate more clearly; Muse Wonder 1.0 and Gemini 4 Argon, which lead on DeepSWE v1.1, land mid-pack here. Note that Cognition's own SWE-2 is on the board, and there is no independent reproduction yet.
What developers can take from it
When picking a model for a coding agent, this leaderboard is closer to everyday experience than spec-following benchmarks because it penalizes missed fixes, unrelated edits and weak tests. Cognition also found that higher reasoning effort does not always help, since models write more code and touch more files, which matches what many developers see; it is a reason to tune effort per task. The full leaderboard and cost comparison are on Cognition's blog and FrontierCode leaderboard page.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.