Empirical Benchmark Unmasks Sub-Agent Concurrency: SWE-bench Success Drops 24% as Token Costs Surge up to 3.3x

Background and the Problem

The AI developer tooling landscape has heavily hyped multi-agent and sub-agent concurrency as the silver bullet for complex software development. Frameworks claim that delegating tasks to concurrent child agents for parallel searching, coding, and review accelerates turnaround times. However, in distributed computing, concurrency without synchronization triggers race conditions, deadlocks, and resource exhaustion. Until now, the community lacked controlled, large-scale empirical benchmarks measuring the net utility of sub-agent decomposition. Systems researcher @zhou_xiaoma released a comprehensive evaluation across over $20,800 of API invocations, delivering critical insights.

Architecture and How It Works

1. Rigorous Paired Empirical Evaluation:

• Evaluated three state-of-the-art coding systems: Claude Code, Codex, and Kimi Code;
• Executed 2,124 complete coding sessions across 354 tasks, comparing single-agent sequential execution against native sub-agent concurrency under identical prompts and sandboxes;

1. Three Dominant Concurrency Failure Modes:

• Unbounded Fan-Out & Budget Exhaustion: Parent agents aggressively spawn dozens of sub-agents, exhausting context windows and token budgets before implementing fixes;
• Orphaned / Unjoined Deliverables: Parent agents conclude and submit solutions prematurely while long-running sub-agents are terminated before returning results;
• Delivery Race Conditions: Concurrent sub-agents modify overlapping source files, overwriting each other's changes and inducing compilation breaks.

Benchmarks and Measured Results

Based on 2,124 execution traces (Figure 1, Benchmark Disclosures):

• SWE-bench Verified (100 Standard Bug Fixes):
• Claude Code: Single-agent sequential execution solved 83%, which plummeted to 59% with sub-agents enabled (a 24-point drop), while execution time lengthened from 17.5 to 27.8 minutes;
• Kimi Code: Dropped from 77% to 59%;
• Codex: Remained stagnant at 71%;
• Token Inflation: Sub-agent concurrency drove token consumption up to 1.41x to 3.31x that of sequential baselines;
• Long-Horizon Multi-File Challenges (LoopsBench):
• On sprawling multi-repository tasks, sub-agents proved their utility: Claude Code's success rate surged from 7.1% to 21.4%.

Getting Started for Developers

• Source Disclosure: Verified on X via @zhou_xiaoma (status/2112363831284650435);
• Architectural Rules of Thumb:
• Sequential by Default: For 90% of bug fixing and refactoring, maintain a single sequential agent loop to avoid 3x token penalties and a 20%+ accuracy degradation;
• When Sub-Agents Make Sense: Enable concurrency only when tasks have strictly isolated file ownership boundaries, when running parallel speculative attempts (picking the best result), or when deploying read-only independent verifier agents.