Google and the University of Waterloo posted ReproBridge on arXiv (2611.08452, 2026-11-05). It turns the code fixes, environment setup, launchers and guide an agent discovers while reproducing a paper into a standardized artifact. ReproBridge-Bench covers 36 papers, 387 experiments and 1,029 tasks. Author-reported: across Sonnet 4.6, Sonnet 5, Opus 5 and Gemini 3.8 Flash, mean success rises from 76.4% to 92.6% and API spend drops 19.7% on average.
Key Takeaways
- ✓Benchmark: 36 papers, 387 experiments, 1,029 tasks, each anchored by 5 human-verified reference scores
- ✓Author-reported: mean reproduction success 76.4% → 92.6% across 4 models; API spend down 19.7% on average
- ✓Cross-model spread narrows from 21.6 to 3.9 points; all four models exceed 90%
- ✓Gains range from 5.5pp (Opus 5) to 23.2pp (Gemini 3.8 Flash); removing code or environment hurts most in ablations
- ✓24-hour wall-clock limit per paper on 8×H100; Codex serves as the primary Auditor
Key Decision Metrics at a Glance
Turn your technical choice into a development budget
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Core Background & Industry Pain Points
When coding agents reproduce ML papers, the hard part is often not reasoning but the gap between paper and released code: dependency conflicts, missing aggregation scripts, hyperparameters buried in appendices, even bugs. Every fresh session rediscovers the same fixes, which is slow and burns tokens. The Google team asks whether preserving the steps and decisions from a first successful reproduction makes later independent reproductions more reliable and cheaper.
Architecture Highlights & Internals
ReproBridge defines a standardized refined reproducible artifact with four parts: a cleaned codebase, environment setup, experiment launchers and a reproduction guide, built by an agent workflow (Opus 5 builds the artifacts in the main experiment). In downstream evaluation, fresh Reproducer sessions get only the paper, the tasks and either raw code or the refined artifact, never the construction chats or reference answers. A separate Auditor checks execution evidence so agents cannot report numbers without running anything.
Authoritative Benchmarks & Measured Scores
All figures are author-reported. ReproBridge-Bench has 1,029 tasks. Across Sonnet 4.6, Sonnet 5, Opus 5 and Gemini 3.8 Flash, mean success rises from 76.4% to 92.6%, downstream API spend falls 19.7% on average, and the cross-model spread narrows from 21.6 to 3.9 points. The smallest gain is for Opus 5, the strongest raw reproducer (+5.5pp); the largest is for Gemini (+23.2pp). The paper notes vendor harnesses differ, so cross-model comparisons are descriptive.
Developer Hands-on Guide
If your agent keeps failing to run a repo, try this before reaching for a bigger model: capture the environment, patches, launch commands and output mapping from the first successful run as a README-level guide committed to the repo, and let the next agent reuse it. See arXiv 2611.08452 and the official project page for details; validate on your own repositories first and do not treat 92.6% as a general guarantee.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.