Artificial Analysis Ships Endpoint Accuracy Index v1.1: Nearly 1 in 4 of 100+ Serverless Endpoints Score Below Reference

Core Background & Industry Pain Points

The same open weights can lose accuracy when hosted by different API providers through quantization, sampling defaults, context truncation, output caps, or tool-call parsing differences. Teams that shop only on price and speed can unknowingly buy a same-named model with lower effective accuracy. Artificial Analysis Endpoint Accuracy Index turns that gap into a comparable score.

Architecture Highlights & Internals

Each serverless endpoint is rescored against a fixed suite and reported as a percentage of a self-hosted official-weights reference (100%), with equal weights and 95% confidence intervals from repeats. v1.1 upgrades AA-LCR-25 to AA-LCR v1.1 (answer-key fixes; grading via GPT-5.6 Luna medium) and raises repeats from 10 to 25; Artificial Analysis says the average 95% CI tightened from about ±4.7 to ±2.6 points. Coverage expands to six open-weights models: DeepSeek V4.1 Flash 1017 (max), GLM-5.3 (max), Kimi K3 (max), gpt-oss-120b (high), plus new GLM-5.3 Flash (max) and Meta Muse Spark Open (xhigh). Results are dated point-in-time snapshots, not live monitoring.

Authoritative Benchmarks & Measured Scores

Figures below are Artificial Analysis official/vendor-reported. Overall: nearly 1 in 4 of 100+ endpoints score below the reference. GLM-5.3 Flash: 16 of 22 endpoints statistically in line with the reference, one about 104.2; two endpoints capped near 32K output tokens cut off roughly half of HLE-250 answers, scoring about 22% and 29% versus 55% for the reference; other endpoints that produced about 30% fewer reasoning tokens without truncation scored about 48% versus 55% on HLE-250. Muse Spark Open: three of five endpoints about 99.5–99.9 (including FP4); two mainly lose on BFCL-500 (about 50%/51% versus 59%) by returning text-typed tool args (e.g., booking IDs) as numbers. Always re-check methodology and per-model provider pages for the latest snapshot dates.

Developer Hands-on Guide

When picking a provider, use Endpoint Accuracy Index alongside price/speed: prefer endpoints whose confidence intervals overlap the reference; when debugging low scores, inspect output caps, default reasoning effort, tool-call JSON/typing, and context truncation. Self-hosted or first-party endpoints remain useful baselines. See artificialanalysis.ai methodology and provider pages; this index does not replace your own online evals and regression suites.