The paper “An Empirical Study of Harness Design for Coding Agents” (HF papers/2609.20804) keeps a fixed execution loop and varies planning, action space, and context management across four models on SWE-Bench Verified and Terminal-Bench 2.1 (176 matched settings). Key findings: context management grows more valuable as the window shrinks (mostly by preventing overflow failures); rule-based elision before LLM summarization is the strongest efficiency combo; planning acts as an accuracy scaffold for weaker models and matters less for stronger ones.

Key Takeaways

  • ✓Fixed execution loop with ablations of planning, action space, and context management across 176 settings.
  • ✓Evaluated on SWE-Bench Verified and Terminal-Bench 2.1 with four models.
  • ✓Tighter windows raise the value of context management; rule elision then LLM summary wins on efficiency; planning helps weaker models more.
ADSponsored