X Discussion: YC President Garry Tan Open-Sources Harness Evals: 7 Model-Harness Setups All Pass 71-79% of 20 Tasks, With Cost the Real Gap: OpenClaw About $0.44 per Solved Task, Claude Code $0.22, Codex $0.17

Background & Industry Pain Points

When picking a coding agent, attention usually goes to the model, but the same model wrapped in different harnesses (system prompt, tools, memory) can differ a lot in cost and speed. At 18:09 UTC on November 8, 2026, Y Combinator President and CEO Garry Tan posted on X that out of curiosity he is doing open-source eval work on harnesses with frontier models, attaching a screenshot of early results, and then shared the GitHub repo garrytan/g-harness-evals (MIT license). At capture the post had about 312 likes, 84 replies, 117 bookmarks and about 41,767 impressions, and X listed it as a trending discussion. This is a personally run eval, not an official vendor benchmark.

Architecture Highlights & Mechanisms

The repo README describes the method: hold the model (Opus 5.5 or GPT-6.1 Sol) and the tasks fixed and swap only the harness. Every setup runs the same tasks, graders, preamble and metering proxy, and the receipts are in the repo so anyone can regrade. There are 7 setups: Claude Code, OpenClaw, Hermes Agent and QM (Pi loop) on Opus, plus Codex, OpenClaw and Hermes Agent on Sol. Cost per solved task is the setup's whole spend at list price divided by its passes. Claude Code and Codex each run only their own model, so they are never compared with each other.

Authoritative Benchmarks & Measured Scores

All figures are the author's. Pass rate: on Opus, Hermes 78.8%, Claude Code 76.3%, OpenClaw 76.3% and QM 72.5%; on Sol, Hermes 75.0%, OpenClaw 73.7% and Codex 71.3%. The 95% ranges overlap heavily; the screenshot explains that 14 of the 20 tasks scored the same in every setup, so the test was too easy and this is "a test problem, not a tie", with a harder Terminal-Bench rerun in progress. Cost: on Opus, QM $0.19, Claude Code $0.22, Hermes $0.24 and OpenClaw $0.44 per solved task; OpenClaw takes about 99.4 s per trial against 40.1 s for Claude Code. On Sol, Codex is cheapest and fastest at $0.17 and 30.1 s. The README says the cost differences are descriptive with no significance test, QM's network was not held fixed, and both OpenClaw cells are incomplete because the last task hit a cap stop.

Developer Hands-on Guide

To check the numbers yourself, clone garrytan/g-harness-evals and start with docs/scoreboard.md for the full tables and findings/harness-cost for the cost breakdown. The author's current advice is to choose on cost and speed while pass rates can't be separated: Claude Code on Opus, Codex on Sol. These are dev results on 20 public tasks rather than held-out confirmation, so it is worth waiting for the Terminal-Bench rerun before drawing firm conclusions. The repo also discloses conflicts: QM is a YC project, YC is an investor in Hermes, and gstack and gbrain are the author's own projects, so weigh those rows accordingly.