The paper “An Empirical Study of Harness Design for Coding Agents” (HF papers/2609.20804) keeps a fixed execution loop and varies planning, action space, and context management across four models on SWE-Bench Verified and Terminal-Bench 2.1 (176 matched settings). Key findings: context management grows more valuable as the window shrinks (mostly by preventing overflow failures); rule-based elision before LLM summarization is the strongest efficiency combo; planning acts as an accuracy scaffold for weaker models and matters less for stronger ones.

Key Takeaways

  • ✓Fixed execution loop with ablations of planning, action space, and context management across 176 settings.
  • ✓Evaluated on SWE-Bench Verified and Terminal-Bench 2.1 with four models.
  • ✓Tighter windows raise the value of context management; rule elision then LLM summary wins on efficiency; planning helps weaker models more.
🧭

Heavy Claude Code use: compare subscription limits and API bills

Compare 40 dev plans & simulate token costs vs $20/mo subscriptions

Action HubReady to adopt this in production?

Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.

Up NextFactory launches Droid 1.0: programmatic tool calling by default, hundreds of tool calls per model request, leads native harnesses on its own Terminal-Bench 4 runs
Read Now→