On November 18 Ramp Labs, the research arm of fintech company Ramp, published Ramp SWE-Bench v2 on its official X account: 500 tasks built from work that more than 200 Ramp engineers actually shipped to production, with stricter verification and each frontier model running in its native harness (Claude Code, Codex, OpenCode, Grok Build and others). The official leaderboard (pass@1, high effort) puts GPT-6.2 Astra first at 76.4%, Carbon at 75.4%, and Fable 5.2 and GPT-6.2 Sol tied at 73.4%. Mean cost per task ranges from $0.07 for GPT-6 Luna to $4.59 for Muse Wonder.
Key Takeaways
- ✓Task source: 500 tasks drawn from real financial software work that 200+ Ramp engineers shipped to production
- ✓Top five: GPT-6.2 Astra 76.4%, Carbon 75.4%, Fable 5.2 73.4%, GPT-6.2 Sol 73.4%, Hydrogen 71.6%
- ✓Value: GPT-6.1 Terra scores 70.4% at $0.46 per task, Sonnet 5.5 63.2% at $0.47, DeepSeek V4.1 Flash 60.8% at $0.22
- ✓On the hardest 25% of tasks Carbon (43%) and Hydrogen (41%) lead clearly, ahead of Astra at 30% and Fable 5.2 at 14%
- ✓Speed: GPT-6.2 Astra in Codex averages 6.5 minutes per task, while Carbon takes 26.3 minutes and 349 tool calls

Key Decision Metrics at a Glance
Turn your technical choice into a development budget
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Background
Most public SWE-bench style tasks come from open-source repositories, which look very different from real company codebases and are increasingly easy to overfit. Ramp built its benchmark from work its own engineers shipped, to measure how AI coding agents handle real financial product code.
How it works
According to the release, v2 has 500 tasks from production work by more than 200 engineers, with stricter verification than the previous version. Each model runs in its native harness: Claude models in Claude Code, GPT models in Codex, Carbon and Hydrogen in OpenCode, Grok 4.7 in Grok Build. Scores are pass@1 at high effort.
Benchmarks
Official leaderboard: GPT-6.2 Astra 76.4% ($3.31 per task), Carbon 75.4% ($4.25), Fable 5.2 73.4% ($3.79), GPT-6.2 Sol 73.4% ($1.05), Hydrogen 71.6%, GPT-6.1 Terra 70.4%, Opus 5.5 67.8%, Grok 4.7 64.4%.
What it means for developers
If cost matters most, GPT-6.2 Sol and GPT-6.1 Terra get close to the top score at roughly a third to an eighth of the cost. On the hardest tasks Carbon and Hydrogen do better, but take far longer and make many more tool calls. This is one company's codebase, so test on your own project before drawing conclusions.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.