FrontierRefactor Introduces First Code Performance Benchmark: 41% of Agent Patches Pass Tests with Zero Gain or Regressions

Background and the Problem

In industrial software engineering, performance refactoring is a critical discipline. Unlike repairing discrete bug tickets, performance optimization requires restructuring memory layouts, concurrency primitives, or algorithmic bottlenecks while strictly preserving external behavioral contracts. However, existing agent benchmarks (such as SWE-bench, HumanEval, and LiveCodeBench) suffer from a critical limitation: they treat unit test pass rates as the sole measure of success, ignoring runtime efficiency and compute resource consumption. Consequently, when developers instruct agents to optimize codebases, models frequently deliver syntactically pristine patches that pass test suites but trigger catastrophic runtime latency regressions. To resolve this evaluation void, researchers introduced FrontierRefactor in arXiv:2610.25834.

Architecture and How It Works

1. Multi-Scale, Multi-Language Engineering Suite:

• Covers seven major languages (C/C++, Rust, Go, Python, Java, JS) across 60 Amdahl-qualified real-world tasks spanning micro-kernels to full repositories ($10^1$ to $10^4+$ LOC);

1. Dual-Paradigm Refactoring Framework:

• Evaluates in-place optimizations (SIMD vectorization, cache tiling, asynchronous I/O) against cross-language rewrites (e.g. migrating Python hotspots into Rust/PyO3);

1. Reference-Patch-Free Cascade Protocol:

• Combines automated probe distillation with physical clock noise calibration to eliminate micro-benchmarking artifacts and verify authentic speedups across heterogeneous server architectures.

Benchmarks and Measured Results

Evaluations across leading coding agent configurations (Table 1, Figure 4):

• Decoupling of Functional Correctness and Speedup:
• Over 41.2% of candidate patches that successfully passed full functional verification suites failed to produce verified physical speedups;
• 19.3% caused severe performance regressions (degrading runtime by 15% to over 300%), while 21.9% were lost entirely in measurement variance;
• Orthogonal Architectural Bottlenecks:
• Cross-language migrations unlock major theoretical speedups from native compilation, but agents struggle with interface contract fidelity at repository scale;
• In-place optimizations reliably preserve functional invariants, but rarely yield acceleration without runtime profiling hooks;
• The Catalytic Value of Execution-Timing Feedback:
• Equipping agents with multi-turn runtime profiling feedback expanded strict refactoring success rates by 2.4x, proving that execution feedback is indispensable for performance engineering.

Getting Started for Developers

• Paper & Dataset Specification: Accessible via arXiv:2610.25834;
• Engineering Blueprint: Teams deploying coding agents for enterprise refactoring must not rely solely on unit tests as PR merge gates. Harnesses must integrate automated runtime profiling feedback into agent loops to prevent performance regressions from polluting production codebases.