Kaggle and IBM Research launch the ScarfBench Java migration leaderboard: Gemini 4 Argon with Antigravity leads at 75.2%, and about 72% of failures happen at runtime

Background and the problem

Much enterprise software is written in Java, and migrating between frameworks (for example from Jakarta EE to Spring Boot or Quarkus) touches dependency injection, persistence, web routes, build scripts and deployment config, often across dozens of files. Coding agents can easily produce a migration that looks right and even compiles, yet breaks once deployed. Most coding benchmarks score single-file patches or unit tests and rarely check whether a whole application still behaves the same after migration. ScarfBench targets that gap: IBM released the benchmark earlier this year, and on October 30 the official Kaggle account announced a jointly maintained leaderboard.

Architecture and how it works

According to IBM Research's October 27 technical note, ScarfBench evaluates model-plus-harness combinations rather than models alone, so results should be read per combination. Its 204 tasks cover migrations among Jakarta EE, Quarkus and Spring, and each migrated app must build, start and pass behavioral tests. The leaderboard metric is the smoke score, which awards partial credit and weights larger test suites more; all-tests-pass and mean scores are also reported. Kaggle consulted on the design, independently reproduced the runs on its platform, sponsored the compute and will add new models as they ship.

Benchmarks and measured results

All figures below are IBM's. Top smoke scores: Gemini 4 Argon + Antigravity 75.2%, Claude Opus 5.5 + Claude Code 70.5% and Claude Opus 5 + Claude Code 69.8%, followed by GPT-6.1 Sol + Codex 68.4%, Gemini 3.8 Flash + mini-swe 66.0%, GPT-6 Astra + Codex 65.4%, Qwen 3.8 Max + qwen-code 63.7% and Kimi K3 + kimi-code 61.1%. Claude Opus 5.5 has the highest all-tests-pass rate at 72.1%. Eight of 10 configurations compiled at least 97% of tasks, but smoke scores spanned more than 50 points, and about 72% of 745 classified failures happened at runtime. Cost per solved task was $1.25 for GPT-6.1 Sol + Codex and $10.10 for Claude Opus 5.5 + Claude Code; Antigravity, qwen-code and OpenHands reported no cost data.

Practical guide for developers

If you plan to let an agent migrate a framework, the main lesson is that compiling is only the start: add endpoint-level regression tests first, and watch for lost routes, changed HTTP status codes and leftover source-framework config. When choosing a setup, compare the smoke score with the all-pass rate and the cost per task: Opus 5.5 if you want the most fully green migrations, GPT-6.1 Sol + Codex if budget matters. The leaderboard is on Kaggle; public submissions are not open yet, and IBM says they are on the roadmap.