Artificial Analysis Launches Decision Index v2: New Calibration-Aware Decision Score, 45% Private Data, Drex Max v1.1 Narrowly Leads Decision Models at 73

Background and pain points

"Decision models" are small models that skip long-form text and directly return an option plus a probability for classification, extraction, tool-call and allow/deny questions, often as routers and reviewers inside agent pipelines. What matters in production is whether a stated 90% confidence is trustworthy, because systems use confidence thresholds to decide whether to act automatically or escalate to a person or a stronger model. Artificial Analysis v1 mostly measured accuracy and used 75% public data, which invites benchmark gaming. Version 2, released on November 8, rebuilds the index around both issues.

Key design changes

The new Decision Score gives each question 0-100 points: right answers score above 50, wrong answers below 50, and confidence sets the position within each half. Overall it equals 50 x (1 + accuracy - Brier score), the average payoff across all thresholds, so it does not depend on one cutoff. New AA-Paperclip (30% weight) has 1,500 private documents (resumes, receipts, invoices, papers), each with 5 decisions and 10 extractions, and a document only passes if all 15 are right. AA-Symmetry (15%) uses 120 base questions and 440 variations (reordered options, swapped labels, negated questions) to test stability. Private data rises from 25% to 45%.

Benchmark results

All figures are Artificial Analysis results. Drex Max v1.1 (Nace AI) and GLiDE v1.2 (Fastino Labs) both score 73 and Mammoth 27B (Base Labs) 72, yet GLiDE leads on raw accuracy (60 vs 58), so calibration changes the ranking. On AA-Paperclip, Claude Haiku 5.5 (Max) passes 92% of documents; Drex Max v1.1 gets 97% of individual answers right but passes only 63% of documents. Mammoth 27B scores 96 on AA-Symmetry and picks the same answer across variations on 99% of questions.

Practical guidance

If you use small models for routing or review in agents, toggle "Include language models" on the leaderboard to compare cost and speed: Drex Max v1.1 costs $102 per 1M tasks at about 275 ms per task, while GPT-6 Luna (Max) is only about 3 points higher at roughly 15x the cost and 40x the latency. Watch billing too: GLiDE v1.2 re-bills document input for every question, so a list price only 33% higher ends up nearly 8x the total cost of Drex Max. Systems that rely on confidence thresholds should look at the Decision Score rather than accuracy alone.