Benchmark13d ago
FrontierRefactor Introduces First Code Performance Benchmark: 41% of Agent Patches Pass Tests with Zero Gain or Regressions
Published in arXiv:2610.25834, FrontierRefactor establishes the industry's first evaluation benchmark targeting autonomous coding agents on multi-scale, performance-oriented codebase refactoring. Unlike traditional benchmarks like SWE-bench that measure only functional pass rates, FrontierRefactor spans seven programming languages and 60 Amdahl-qualified tasks ranging from kernels to full repositories (10^1 to 10^4+ LOC). Initial evaluations reveal that over 41% of agent patches pass functional equivalence suites yet yield zero verified speedup—with 19.3% triggering severe runtime performance regressions, highlighting the vital need for multi-turn profiling feedback.
Benchmark4d ago
Kaggle and IBM Research launch the ScarfBench Java migration leaderboard: Gemini 4 Argon with Antigravity leads at 75.2%, and about 72% of failures happen at runtime
Kaggle announced ScarfBench with IBM Research, a 204-task benchmark that tests whether coding agents can migrate enterprise Java apps between Jakarta EE, Quarkus and Spring without changing behavior. IBM's October 27 technical note shows Gemini 4 Argon with Antigravity first on the smoke score at 75.2%, Claude Opus 5.5 with Claude Code highest on all-tests-pass at 72.1%, and GPT-6.1 Sol with Codex cheapest at $1.25 per solved task.
Research40d ago
Who Finishes the Job? First Empirical Study Reveals AI Coding Agent Merges Require 1.62x More Follow-Up Fixes
Researchers from Monash University and the University of Melbourne published the first large-scale empirical study tracking the post-merge lifecycle of AI coding agents across GitHub open-source repositories (arXiv:2609.26847). Analyzing 6,774 merged agent PRs (spanning Devin, Cursor, Claude Code, GitHub Copilot, and OpenAI Codex) against 5,044 contemporaneous human PRs across repos with 500+ stars, the study reveals that merged agent code requires verified follow-up bug fixes at 1.62 times the odds of human merges. However, 69.6% of verified fixes are authored by the same agent, and 76.4% of fix PR commits remain entirely agent-driven, debunking assumptions that human maintainers bear the full manual burden of cleanup.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.