Artificial Analysis Releases HealthBench & MLCR-AA: Exposing Severe Verbosity Bias in Medical AI Rankings

Background and the Problem

As frontier AI models are integrated into clinical triage, medical record synthesis, and differential diagnosis, rigorous evaluation of healthcare reasoning is critical. However, modern automated evaluation pipelines rely extensively on LLM-as-a-Judge frameworks. This paradigm carries a systemic, unaddressed flaw: Verbosity Bias. Automated judges consistently award higher scores to longer, structurally elaborate answers regardless of medical conciseness or clinical necessity. In high-stakes emergency room triage, bloated answers that bury vital warnings beneath repetitive prose constitute active patient safety risks. To expose and eliminate this distortion, independent evaluation platform Artificial Analysis released HealthBench Professional alongside the length-adjusted MLCR-AA benchmark (@ArtificialAnlys).

Architecture and How It Works

1. HealthBench Professional Clinical Protocol:

• Spans complex differential pathology, pharmacovigilance, contraindication detection, and guideline compliance across 2,000+ real-world clinical patient cases;

1. Length-Adjusted MLCR-AA Normalization:

• Decouples raw word count from diagnostic efficacy by implementing a marginal information density metric per token;
• Penalizes repetitive boilerplate explanations that fail to introduce distinct, verified clinical assertions;

1. Information Density Auditing:

• Directly measures how effectively models distill actionable insights under ambiguity without burdening physicians with redundant text.

Benchmarks and Measured Results

The comparative findings revealed dramatic ranking inversions (verified via official benchmark telemetry):

• Raw HealthBench Scoring (Distorted by Verbosity):
• Anthropic model generations were found to be 4–6x longer on average than competing models;
• Buoyed by expansive length, Claude Opus 5.5 scored 71.4%, holding #1 ahead of Sonnet 5.5 (70.8%) and GPT-6 Astra (69.7%);
• Length-Adjusted MLCR-AA Scoring (Normalized Ground Truth):
• When length bias was systematically neutralized, GPT-6 Astra (max) and Claude Sonnet 5.5 tied for the #1 position globally;
• Claude Opus 5.5 plummeted from #1 down to #6, proving that its raw lead was largely an artifact of verbose, dilutive formatting rather than superior medical reasoning.

Getting Started for Developers

• Benchmark Access: Available on Artificial Analysis (https://artificialanalysis.ai);
• Clinical AI Blueprint: Healthcare engineering teams building medical copilots must reject uncalibrated raw LLM benchmarks. SFT and RLHF pipelines must incorporate strict brevity and density penalties to ensure clinical recommendations remain concise, actionable, and free of deceptive filler.