HERA: Harness-Environment Co-Evolution for Reliable Agentic Abstention Scoring 83.3% on Held-Out Tasks

Background and the Problem

Autonomous agents deployed in executable software environments excel at sequential tool execution when a clear solution exists. However, when faced with infeasible objectives—such as missing database tables, ungranted permissions, or logically impossible constraints—agents routinely demonstrate catastrophic overconfidence. Instead of recognizing dead-ends and politely abstaining, they hallucinate synthetic outputs, cycle through repetitive tool calls, or execute unintended destructive operations. Standard mitigations tune system prompts against static evaluation suites; however, static benchmarks quickly saturate and fail to anticipate novel failure modes in unpredictable production workflows. Researchers introduced HERA (arXiv:2610.06563) to establish reliable agentic abstention.

Architecture and How It Works

1. Harness-Environment Co-Evolution:

• Replaces static benchmarking with a dynamic, dual-evolutionary loop;
• An environment-mutation pipeline systematically generates strictly verified pairs of feasible and infeasible tasks in live runtime sandboxes;

1. Failure-Driven Scaffold Adaptation:

• As the agent harness encounters unhandled edge cases or hallucinatory retries, execution traces trigger automated scaffold refinement, enhancing failure recognition heuristics;

1. Broad Multi-Model Transferability:

• Although seeded using GPT-5.6-Luna, the resulting harness transfers zero-shot across 19 other commercial and open foundation models without architecture-specific retraining.

Benchmarks and Measured Results

Evaluated on the newly introduced hera-bench across hundreds of complex tool-use tasks (Table 1, Figure 3):

• Agentic Abstention Success:
• On held-out tasks, baseline agent harnesses correctly abstained in only 61.7% of infeasible instances;
• HERA-evolved harnesses boost abstention fidelity to 83.3%;
• Paired Success Rate:
• On matched pairs (where an agent must solve the feasible version and abstain from the infeasible counterpart), success surged from 48.3% to 70.0%;
• Cost and Execution Overhead:
• Analysis across 14 models demonstrated that preventing fruitless multi-turn looping reduced average token consumption and API costs by over 34%.

Getting Started for Developers

• Paper & Dataset: Detailed methodology and benchmark suites are accessible via arXiv:2610.06563;
• Engineering Principles for Enterprise Agents: Teams deploying autonomous SRE, financial auditing, or database management agents must adopt paired feasibility stress-testing, implementing explicit abstention policies in runtime scaffolds to prevent destructive rogue actions.