HF's Joël Niklaus Ships Harness Leaderboard: 28,849 Trials Across 9 Coding Agent Harnesses; Author Reports mini-swe-agent ~51.4% Average
On 2026-11-05, Hugging Face Harness Optimization researcher Joël Niklaus (@joelniklaus) released the Harness Leaderboard: same model, provider, and reasoning effort with only the harness swapped, covering GLM-5.3-Flash, DeepSeek-V4.1-Flash, and Flint on Terminal-Bench 2.1 and DeepSWE—28,849 trials and about $4,746 of inference. Author-reported averages: mini-swe-agent ~51.4% resolve rate; Pi ~49.4% at about $0.29 per solved task. This is an individual research leaderboard, not a Hugging Face company product announcement.
Key Takeaways
- Source: personal post by HF Harness Optimization researcher Joël Niklaus on 2026-11-05T16:04:08Z—not a company announcement
- Scale (author): 9 harnesses on open models, 28,849 trials, ~$4,746 inference spend
- Method: pin model/provider/reasoning effort; 3 attempts per task; Terminal-Bench 2.1 and DeepSWE
- Results (author): averaged mini-swe-agent ~51.4%, openclaw ~27.3%; Pi ~49.4% at ~$0.29 per solved task
- Use: public HF Space leaderboard + trajectories; Harness Lens documents prompts/tools for each harness
