On October 19 independent evaluator Artificial Analysis launched two health-conversation leaderboards built on OpenAI's HealthBench (5,000 realistic health conversations) and HealthBench Professional (525 clinician conversations), graded against physician-written rubrics and reported as both raw and OpenAI-style length-adjusted scores. On Professional, GPT-6 Astra (max) leads at 66.0% length-adjusted, ahead of GPT-6.1 Sol (65.1%) and Claude Sonnet 5.5 (63.0%); Claude Opus 5.5 has the top raw score (71.4%) but drops to sixth (59.9%) because its ~5,900-character answers are penalized. On HealthBench, Claude Sonnet 5.5 leads at 62.1%, with DeepSeek V4.1 Flash close behind at 60.9% for about 1/16 of the cost per task.
Key Takeaways
- ✓HealthBench Professional (length-adjusted): GPT-6 Astra (max) 66.0%, GPT-6.1 Sol (max) 65.1%, Claude Sonnet 5.5 63.0%.
- ✓Claude Opus 5.5 posts the highest raw score (71.4%) but averages ~5,900 characters per answer, nearly 3x the 2,000-character threshold, and falls to sixth at 59.9%; Claude Fable 5.1 also loses over 11 points.
- ✓The Professional score-vs-cost Pareto frontier is all OpenAI: GPT-6 Luna (max) scores 61.9% at $0.004 per task, under 2% of GPT-6 Astra's $0.27.
- ✓HealthBench: Claude Sonnet 5.5 leads at 62.1%, DeepSeek V4.1 Flash 60.9% at $0.005 per task, GPT-6.1 Sol 58.5%.
- ✓Grading follows OpenAI's original setup: GPT-5.4 (low) grades Professional and GPT-4.1 grades HealthBench.
Key Decision Metrics at a Glance
Turn your technical choice into a development budget
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Background
LLMs are increasingly used for health questions, but general benchmarks do not measure whether an answer is accurate, complete and clearly communicated in clinical settings. OpenAI's HealthBench grades replies against physician-written rubrics; Artificial Analysis has now turned it and the harder HealthBench Professional into public, continuously updated leaderboards.
How it works
HealthBench has 5,000 conversations with everyday users and health workers across seven themes, with each rubric item tagged to one of five axes: accuracy, completeness, communication quality, context awareness and instruction following. HealthBench Professional has 525 clinician-only conversations covering consultation, writing and research, including adversarial cases. The leaderboards report raw scores and OpenAI's length-adjusted score, which penalizes replies over 2,000 characters and boosts shorter ones. Grading follows OpenAI's setup: GPT-5.4 (low) for Professional and GPT-4.1 for HealthBench.
Benchmark results
All figures are Artificial Analysis'. Professional, length-adjusted: GPT-6 Astra (max) 66.0%, GPT-6.1 Sol 65.1%, Claude Sonnet 5.5 63.0%; Claude Opus 5.5 has the top raw score (71.4%) but drops to sixth at 59.9%. HealthBench: Claude Sonnet 5.5 62.1%, DeepSeek V4.1 Flash 60.9%, GPT-6.1 Sol 58.5%. GPT-6 Luna scores 61.9% on Professional at just $0.004 per task.
Getting started
If you build health Q&A, use the Artificial Analysis HealthBench pages to shortlist models by score and cost per task. Note that the graders are OpenAI models and the length adjustment reshuffles rankings: use raw scores if your product needs thorough answers and adjusted scores if brevity matters. The eval code is in OpenAI's simple-evals repo; re-test on your own data.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.