Opera: A Verbal Critic Framework for Long-Horizon Coding Agents Tracks Diagnosed Flaws Until Resolution to Boost SWE-Bench by 15%

Background and the Problem

As coding agents scale from single-turn code generation to long-horizon repository issue resolution (e.g. SWE-Bench), trajectories routinely exceed dozens of complex environment interactions. Autonomous agents require timely corrective signals to prevent cascading failures. However, existing verbal critics exhibit two fatal design flaws: they deliver ephemeral suggestions without following up on downstream trajectory actions, and they frequently issue unverified hallucinations that derail productive environment exploration. Furthermore, policies distilled from single-harness trajectories suffer catastrophic degradation when ported across heterogeneous runtimes.

Architecture and How It Works

Researchers from Rutgers and Lehigh University present Opera (arXiv:2609.33987), an open-source verbal critic architecture designed for long-horizon agents:

1. Persistent Note & Resolution Tracking: Represents every diagnosed defect as a persistent state note. Opera monitors downstream steps to differentiate superficial adherence (e.g. altering comments or adding dummy try-catch blocks) from genuine root-cause resolution.
2. Hybrid Review Triggers: Combines periodic review cadence with event-driven execution triggers (e.g. build tool crashes or syntax errors) to minimize token overhead while intervening precisely at inflection points.
3. Typed Operators & Pre-Delivery Evidence Auditing: Employs typed diagnostic operators and validates proposed feedback against visible terminal and compiler evidence before exposing it to the agent.
4. Approximately On-Policy Rollout Distillation: Captures error-correction trajectories during guided inference to synthesize high-quality training rollouts for offline fine-tuning.

Benchmarks and Measured Results

Benchmarked across four policy models and three demanding software engineering suites:

1. Consistent Benchmark Domination: As a test-time critic, Opera improves baseline resolution rates by 12.4 percentage points on Terminal-Bench 2.1, 15.0 percentage points on SWE-Bench Pro, and 8.9 percentage points on DeepSWE v1.1.
2. 10.2% Gain on Held-Out SWE-Bench Repos: Fine-tuning Qwen3.5-9B on Opera-guided rollouts yields a 10.2 percentage point resolution increase on unseen SWE-Bench Pro repositories without invoking any critic at test time.
3. Harness Invariance: Preserves model performance when transitioning between radically different execution scaffolding (OpenHands to Terminus-2), where standard supervised models suffer severe degradation.

Getting Started for Developers

Opera is open-sourced at GitHub (dongyuanjushi/Opera). Teams building autonomous DevOps agents or self-healing CI/CD bots should abandon ephemeral single-prompt critics. Implementing persistent note state machines inside the orchestration harness ensures that diagnosed software bugs are tracked until deterministic unit tests pass, producing robust self-correcting agents with minimal operational cost.