Researchers from Fudan University, Alibaba (Alibaba Token Hub) and other institutions published Co-Design Bench on arXiv (2610.35138, submitted 2026-10-26), and from November 8 it spread widely on X. It takes 273 web-app revision tasks that real users had to request repeatedly, drawn from 26,314 real coding sessions and 37,463 deduplicated queries (100 multimodal editing tasks and 173 repository editing tasks), each paired with hidden expert-written acceptance criteria, 2,331 in total. Ten models were run in the same Claude Code harness. The best, GPT-6 Astra, met 69.9% of criteria per task on average but passed every criterion on only 26.4% of tasks; Claude Opus 5 reached 18.3% and Grok 4.6 just 6.6%. The paper tested September-era models, so newer ones such as Opus 5.5 and GPT-6.1 are not included.
Key Takeaways
- ✓Scale: 273 tasks (100 multimodal editing plus 173 repository editing) from 26,314 real coding sessions, with 2,331 hidden acceptance criteria
- ✓Top model GPT-6 Astra: meets 69.9% of criteria on average (Strict 0.699) but fully passes only 26.4% of tasks
- ✓Other full-pass rates (paper figures): Opus 5 18.3%, Qwen3.8-Max 16.5%, Kimi K3 15.8%, Qwen3.8-27B 15.0%, GPT-5.6 Sol 14.3%, Grok 4.6 6.6%
- ✓For contrast, the authors measured Opus 4.8 at 99.4% graded completion in agentic mode on the older WebCompass-edit benchmark, a sign that older tests are near saturation
- ✓All ten models score lower on repository edits that span three or four change types than on edits spanning one or two
Key Decision Metrics at a Glance
Heavy Claude Code use: compare subscription limits and API bills
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Background and the Problem
Coding agents keep scoring higher on older benchmarks, yet developers often feel the work is "almost there". The paper gives a contrast: under the same grading pipeline, Opus 4.8 reaches 99.4% graded completion in agentic mode on the older WebCompass-edit benchmark. Co-Design Bench instead uses revision requests that real users had to repeat or roll back, to measure the gap between how much gets done and whether everything gets done.
Design and Mechanism
There are two tracks: Multimodal Editing (100 tasks, optionally with image, audio, video or document attachments) and Repository Editing (173 tasks with coordinated changes across multi-file frontend projects). There is no gold code. A judge-then-refute protocol first grades each criterion using source and browser evidence, then a separate LLM verifier tries to refute every positive verdict, with extra audits for visual and interaction claims.
Benchmark Results
All models ran in the same Claude Code harness with a three-hour cap per task. Full-pass rates (paper figures): GPT-6 Astra 26.4%, Opus 5 18.3%, Qwen3.8-Max 16.5%, Kimi K3 15.8%, Qwen3.8-27B 15.0%, GPT-5.6 Sol 14.3%, Opus 4.8 10.3%, Grok 4.6 6.6%. Qwen3.8-27B ranks second on repository edits. Caveats: grading relies on LLM judges and several authors work at Alibaba; the models tested are September versions, and no scores have been published for newer ones such as Opus 5.5 or GPT-6.1.
Getting Started
The dataset is on Hugging Face (Wiselnn/Co-Design-Bench) and the evaluation code is on GitHub. The practical lesson: judge agent output by whether every requirement passes, not by the share completed; break requests into checkable acceptance criteria, and have a human review changes that span several components or pieces of state.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.