On October 31, independent evaluator Artificial Analysis released AA-Omniscience v1.1, its knowledge and hallucination benchmark and part of Intelligence Index v5. The question set shrinks from 6,000 to 3,000, dropping previously public and saturated questions, fixing errors and ambiguities, and adding new ones, and a new 300-question public set on Hugging Face is calibrated to the private set's difficulty, with 0.99 Pearson and Spearman correlation on Net Score. Gemini 4 Carbon (high) leads with a Net Score of 54, ahead of Claude Opus 5.5 at 48 and Claude Fable 5.1 at 46; Fable 5.1 has the highest accuracy (68%) and Gemini 4 Argon the lowest hallucination rate (4%). Models score about 1.9 points lower than on v1 on average, and MiniMax M3.1 Flash drops 22 points. All figures are as reported by Artificial Analysis.
Key Takeaways
- ✓The question set shrinks from 6,000 to 3,000 across 6 domains and 42 topics: 600 each for science/engineering/math and software engineering, 450 each for finance, health, humanities/social sciences and law
- ✓A new 300-question public set is on Hugging Face, with 0.99 correlation to private-set Net Score and a median difference of 0.0 points
- ✓Net Score ranking: Gemini 4 Carbon (high) 54, Claude Opus 5.5 48, Claude Fable 5.1 46; the best open-weights model, Muse Spark Open, trails Gemini 4 Carbon by 28 points
- ✓Claude Fable 5.1 has the highest accuracy (68%), followed by Claude Opus 5.5 (66%); Gemini 4 Argon has the lowest hallucination rate (4%)
- ✓Rank order keeps a 0.98 Spearman correlation with v1, but scores are 1.9 points lower on average; MiniMax M3.1 Flash falls from 16 to -6 and MiMo V2.6 Pro from 8 to -5 (vendor-reported)
Key Decision Metrics at a Glance
Turn your technical choice into a development budget
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Background
Confident wrong answers remain one of the biggest risks of deploying LLMs. Most leaderboards count correct answers and do not separate a model that guesses from one that admits it does not know. AA-Omniscience is Artificial Analysis's own benchmark; its Net Score rewards correct answers, penalizes wrong ones and treats abstentions neutrally, on a scale from -100 to 100. Some v1 questions had become public or saturated, so v1.1 ships with Intelligence Index v5.
How it works
v1.1 cuts the set from 6,000 to 3,000 questions, removing public and saturated items, fixing errors and ambiguities, and adding new questions, still across 6 domains and 42 topics with 600 software engineering questions. A new 300-question public set is calibrated to the private set's difficulty; at release the two have 0.99 correlation on Net Score with a median difference of 0.0 points. Hallucination rate is the share of incorrect answers among all non-correct responses.
Benchmarks
As reported by Artificial Analysis: Gemini 4 Carbon (high) leads at 54, followed by Claude Opus 5.5 at 48 and Claude Fable 5.1 at 46; Muse Spark Open is the best open-weights model, 28 points behind. Fable 5.1 has the highest accuracy (68%) and Gemini 4 Argon the lowest hallucination rate (4%). Rankings keep a 0.98 correlation with v1, but scores average 1.9 points lower, and MiniMax M3.1 Flash drops 22 points. Artificial Analysis says the drop reflects the dataset change, not model regression.
Getting started
Download the 300-question public set from Hugging Face and run it through your own prompts and tooling to see whether your model abstains when it does not know. For fact-sensitive work such as RAG, support, legal or health, look at accuracy and hallucination rate separately and pick models by domain score rather than the overall ranking. Full results and methodology are on the Artificial Analysis Omniscience page.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.