Benchmark7d ago
Cognition launches FrontierCode 2.0 coding benchmark: it asks whether a PR should be merged, and top model Claude Opus 5.5 scores just 45.5%
Cognition, the maker of Devin, released FrontierCode 2.0 on Oct 27. It keeps the method of maintainer-written tasks graded on whether the code is actually mergeable, but draws tasks from more repositories, adds security patches and performance work, uses deliberately underspecified prompts and adds many thoroughness criteria. In Cognition's results Claude Opus 5.5 leads at 45.5%, GPT-6 Astra follows at 42.8%, and Grok 4.7 and Gemini 4 Argon drop into the 20s, leaving the benchmark far from saturated.
Benchmark15d ago
Devin Fusion with an Opus 5.5 lead and GPT-6 Luna sidekick tops the Artificial Analysis Coding Agent Index at 68.1, at what Cognition says is 31% lower cost per task than Claude Code with Opus 5.5
Cognition said on October 19 that Devin Fusion, pairing Claude Opus 5.5 as the lead with GPT-6 Luna as the sidekick, scored 68.1 on the Artificial Analysis Coding Agent Index v1.5. Artificial Analysis's current data puts it first, but only 0.1 ahead of Muse Code with Muse Wonder Full at 68.0, effectively a tie; Antigravity CLI with Gemini 4 Argon is third at 63.8. Cognition says the pairing costs 31% less per task than Claude Code with Opus 5.5 (vendor-reported). Fusion ships in Devin CLI and Desktop.
Agents24d ago
Devin now works with any personal ChatGPT plan: Go, Plus or Pro subscriptions cover GPT model usage instead of Devin quota
On Oct 9 Cognition said Devin can now link any personal ChatGPT plan (Go, Plus or Pro). The GPT share of each session is billed to the ChatGPT subscription instead of Devin quota or on-demand credits, across Devin Cloud, Desktop and CLI. It requires a self-serve Devin Pro, Max or Teams plan (not Free or Enterprise), and falls back to Devin quota when the ChatGPT allowance runs out.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.