On October 31, independent evaluator Artificial Analysis released AA-Omniscience v1.1, its knowledge and hallucination benchmark and part of Intelligence Index v5. The question set shrinks from 6,000 to 3,000, dropping previously public and saturated questions, fixing errors and ambiguities, and adding new ones, and a new 300-question public set on Hugging Face is calibrated to the private set's difficulty, with 0.99 Pearson and Spearman correlation on Net Score. Gemini 4 Carbon (high) leads with a Net Score of 54, ahead of Claude Opus 5.5 at 48 and Claude Fable 5.1 at 46; Fable 5.1 has the highest accuracy (68%) and Gemini 4 Argon the lowest hallucination rate (4%). Models score about 1.9 points lower than on v1 on average, and MiniMax M3.1 Flash drops 22 points. All figures are as reported by Artificial Analysis.
Zoom
Key Takeaways
The question set shrinks from 6,000 to 3,000 across 6 domains and 42 topics: 600 each for science/engineering/math and software engineering, 450 each for finance, health, humanities/social sciences and law
A new 300-question public set is on Hugging Face, with 0.99 correlation to private-set Net Score and a median difference of 0.0 points
Net Score ranking: Gemini 4 Carbon (high) 54, Claude Opus 5.5 48, Claude Fable 5.1 46; the best open-weights model, Muse Spark Open, trails Gemini 4 Carbon by 28 points
Claude Fable 5.1 has the highest accuracy (68%), followed by Claude Opus 5.5 (66%); Gemini 4 Argon has the lowest hallucination rate (4%)
Rank order keeps a 0.98 Spearman correlation with v1, but scores are 1.9 points lower on average; MiniMax M3.1 Flash falls from 16 to -6 and MiMo V2.6 Pro from 8 to -5 (vendor-reported)
Voice model company Cartesia launched Managed Agents on Oct 29, an end-to-end voice agent platform for enterprise phone calls that tunes its own speech-to-text and text-to-speech models together with the agent harness, and bundles telephony, knowledge bases, tool calls and outbound batches. Cartesia says it ranks #1 on Artificial Analysis's Speech Agent Arena with a 1115 Elo and scores 87.4% accuracy and 90.7% experience on ServiceNow's EVA-Bench (vendor-reported, with GPT-5.5 as the reasoning model). Launch pricing is 50% off at $0.03 per minute.
Zoom
Key Takeaways
On Artificial Analysis's Speech Agent Arena blind preference test, Managed Agents ranks #1 of 30 agents with a 1115 Elo, ahead of Gemini 3.1 Flash Live Minimal (1099), Grok Voice 3.0 (1064) and GPT-Live-1 (1041) (Cartesia blog, as cited by the vendor)
First-attempt results on ServiceNow EVA-Bench across 200+ customer service, IT service and HR scenarios: 87.4% accuracy and 90.7% experience, versus 73.7%/71.2% for Grok Voice 3.0 and 59.0%/61.6% for GPT Realtime 2.1 (EVA-Bench)
All these benchmark runs use GPT-5.5 as the reasoning model; Cartesia's internal evaluation says 85% of calls had no turn-taking issues, versus 67% for other agents
Launch pricing is 50% off at $0.03 per minute; Cartesia says the platform has already handled tens of millions of conversations in finance, retail, hospitality and healthcare
It supports 44 languages; English, Spanish, Hindi, French and Japanese can switch mid-call today, with the rest coming soon
At GitHub Universe 2026 on Oct 28, GitHub made Copilot Fusion generally available. It merges the multi-model orchestration of the Project HydraFusion research preview with Copilot auto model selection: users pick Copilot Fusion and an effort level (Standard, High or Max) in the model picker, and the system decides which models to use and whether to run a Single, Cascade or Critique workflow. GitHub says Fusion Max scored 60 on the Artificial Analysis Coding Agent Index, two points below Claude Code with Opus 5.5 Max, at about half the API cost per task, about half the tokens and about 40% less time. Each underlying model is billed at its standard rate, and paid plans get 30% off through Nov 30.
Zoom
Key Takeaways
Scores (Artificial Analysis Coding Agent Index v2.0): Fusion Max 60 versus 62 for Claude Code with Opus 5.5 Max; $5.59 per task versus $11.42, while the Devin Fusion pairing scores 62 at $8.00
Breakdown: a slight lead on Terminal-Bench 4.0 at 71, third on FrontierCode at 51 (Claude Code 56, Devin Fusion 54), and a weaker 57 on SWE-Atlas-QnA-AA
Efficiency: about 7.2M tokens per task, roughly half of Claude Code with Opus 5.5 and 30% fewer than Devin Fusion; GitHub reports about 40% less time per task
The pool includes Claude Opus 5.5, Claude Sonnet 5.5, GPT-6 Astra and GPT-6 Sol; admins can disable individual models for Fusion only and set Fusion as the default
Availability: all Copilot plans in VS Code, the Copilot app, Copilot CLI and JetBrains IDEs; existing auto, HydraFusion and HydraFusion Max selections migrate to matching effort levels
Google DeepMind announced Gemini 4 Carbon on Oct 28, calling it its strongest model, just two weeks after Gemini 4 Argon. The official blog says it ranks first on v5.0 of the Artificial Analysis Intelligence Index at 59 and leads three cybersecurity evaluations: 88.2% on SEC-bench Pro for finding vulnerabilities, 54.8% on ExploitGym for exploitation and 71.0% on CWE-bench for patching; it also gains 89 points to rank fourth on WebDev Arena at 1769. Access starts this week for a subset of cyber defenders in the Fairwind program, with developers, businesses and consumers to follow. Introductory pricing matches Argon at $2 per million input tokens and $10 per million output tokens, rising to $4 and $20 afterwards.
Zoom
Key Takeaways
Positioning: Google calls it its strongest model, aimed at cybersecurity and long-horizon agentic work; internal teams saw it run for days to root-cause and patch subtle bugs across large codebases and low-level systems
Official scores (evaluated without guardrails): 88.2% on SEC-bench Pro, 54.8% on ExploitGym and 71.0% on Collinear's CWE-bench; 59 and first place on the Artificial Analysis Intelligence Index v5.0; +89 to fourth place on WebDev Arena at 1769
Safety and alignment: successful cheating on Terminal-Bench 4.0 fell to 0.0% with a 0.4% attempt rate, high-severity misalignment alerts in internal live coding dropped by almost 80%, unsupported claims are about half of Argon's, and the Gray Swan indirect prompt injection attack success rate fell from Argon's 0.7% to 0.0%
Internal examples: more than 2x Argon's success rate at generating secure code, and the Chrome team reports more than 10x better high-quality patches than another lab's large frontier model
Rollout: a subset of cyber defenders in the Fairwind program first, alongside the U.S. government's voluntary pre-release testing; no developer API yet; introductory pricing of $2 / $10 per million tokens with cached input 95% off, then $4 / $20
NVIDIA AI announced Nemotron 3.5 Transcribe on Oct 28, with weights on Hugging Face and ModelScope. It is an offline, instruction-driven speech model whose single checkpoint covers transcription in 40 languages, translation between 31 languages and English, speaker attribution in 13 languages, and audio question answering, on recordings up to an hour long with up to eight labeled speakers. NVIDIA says it ranks first among open-weight models on the Hugging Face Open ASR English leaderboard at 5.33% average WER, and Artificial Analysis measured a 2.8% AA-WER, tied for first among open models with Voxtral Small, which uses 24B active parameters. The weights, the Granary v2 training data and fine-tuning recipes are all released.
Zoom
Key Takeaways
Architecture: a speech encoder paired with a Mamba-2/Transformer hybrid MoE language model, about 30B total parameters (34B in the model card counting the audio encoder) and about 4B active per token (4.3B in the model card), shipped in BF16, FP8 and NVFP4
Capabilities: transcription in 40 languages, translation between 31 languages and English, speaker attribution in 13 languages, audio Q&A (trained in English only) and word-level timestamps, on recordings up to 1 hour with up to 8 speakers
Official and third-party figures: first among open models on the Open ASR English leaderboard (5.33% average WER), 4.62% mean WER across 7 languages and 10.98% on long-form; Artificial Analysis AA-WER 2.8% and 1.8% on its AA-AgentTalk voice-agent set (next-best open model 2.3%)
Artificial Analysis measured DeepInfra's hosted endpoint at a 75x speed factor, transcribing a minute of audio in under a second, priced at $2.00 per 1,000 minutes
Licensed under OpenMDW-1.1 for commercial use; Granary v2 data and NeMo Speech fine-tuning tools are released too, and in NVIDIA's medical case study fine-tuning cut WER a further 45.8% relative to the best prompt
On Oct 28 independent evaluator Artificial Analysis released Intelligence Index v5, its biggest revision since v4 in January: it adds Cognition's FrontierCode 2.0 and Terminal-Bench-Science, swaps Humanity's Last Exam for HLE-Diamond, and now draws 60% of its weight from private or held-out test sets. Claude Opus 5.5 (max) leads at 58, ahead of GPT-6 Astra at 57 and GPT-6.1 Sol at 55; the top open-weights model, Kimi K3.1 Pro (weights not yet released), scores 46, widening the gap from 3 points in v4.3 to 12. It also launched Progress Index v1 for comparing models across years.
Zoom
Key Takeaways
Overall (vendor-reported by Artificial Analysis): Claude Opus 5.5 (max) 58, GPT-6 Astra (max) 57, GPT-6.1 Sol (max) 55; Anthropic and OpenAI hold the top five, with Gemini 4 Argon (high) next at 50
The open-weights gap grows from 3 to 12 points: Kimi K3.1 Pro scores 46, while released-weight leaders DeepSeek V4.1 Flash 1017 and GLM-5.3 (max) score 32, below Claude Haiku 5.5 (max) at 36
Cost per task varies more than 13x: among the top 10, GPT-6.1 Terra (max) is cheapest at $0.96 and Claude Fable 5.1 (max) priciest at $13.04; GPT-6.1 Sol trails Opus 5.5 by 4 points at about one eighth the cost ($1.27 vs $10.04)
Four categories at 25% each: Knowledge Work, Coding (FrontierCode 2.0 15%, Terminal-Bench 4.0 10%), Science and General; GDPval-AA v2.1, SciCode and AA-LCR were removed
Tasks are heavier: median output tokens per task rose from 35k to 59k, and effort matters more, with Opus 5.5 scoring 35 at low effort vs 58 at max for 10x the cost
On Oct 27 Ant Ling (inclusionAI) open-sourced Ling-3.1-flash-1019, a further post-trained upgrade of the early-October Ling-3.1-flash: 560B total parameters, 25B active per token, a 1M-token context and an MIT license. Ant says it scores 44 on the Artificial Analysis Intelligence Index v4.3.2 (up from 41), with clear gains on agent tasks such as AA-Briefcase and AutomationBench-AA. It also released Ling-3.1-flash-base, an open foundation checkpoint for further training.
Zoom
Key Takeaways
Scale: 560B total and 25B active parameters, 1M-token context, MIT license, on Hugging Face and ModelScope
Artificial Analysis Intelligence Index v4.3.2: 44, up from 41 for Ling-3.1-flash (vendor-reported)
AA-Briefcase rises from about 1400 to 1524 Elo and AutomationBench-AA from 62% to 68% (vendor-reported)
Science: CritPt from 18% to 26% and SciCode from 54% to 57% (vendor-reported)
Architecture: hybrid-linear MoE with 63 KDA layers plus 9 gated MLA layers (7:1), 512 routed experts and 8 active per token
On 2026-10-23 Artificial Analysis launched Coding Agent Index v2.0, which scores model-plus-harness combinations across 290 tasks with three equally weighted benchmarks: FrontierCode v1.1 from Cognition (100 tasks on real open-source repos, graded on whether a maintainer would merge the change) replaces DeepSWE v1.1, SWE-Atlas-QnA becomes the stricter SWE-Atlas-QnA-AA, and Terminal-Bench 4.0 is unchanged. Claude Code with Claude Opus 5.5 (max) leads at 61.7, ahead of Sonnet 5.5 at 59.3 and Muse Wonder Full at 58.9. Scores fall across the board: the 21 configurations measured on both versions drop from a mean of 54.8 to 47.4.
Zoom
Key Takeaways
Top of the index: Claude Code with Claude Opus 5.5 (max) 61.7, Claude Sonnet 5.5 (max) 59.3, Muse Code with Muse Wonder Full (max) 58.9, Claude Fable 5.1 (max with fallback) 57.6 (Artificial Analysis figures)
On FrontierCode v1.1, Opus 5.5 scores 56.0%, ahead of Codex with DeepSeek V4.1 Flash at 52.6%; 15 of 24 configurations sit between 43% and 49%
Version shift: 21 configurations average 54.8 on v1.5 and 47.4 on v2.0, with 20 of 21 moving down; the same set averages 45.1% on FrontierCode versus 65.6% on DeepSWE
Four points on the cost frontier: DeepSeek V4.1 Flash 47.3 at $0.39 per task, GPT-6.1 Sol 54.1 at $1.50, GPT-6 Astra 54.7 at $7.04, Opus 5.5 61.7 at $11.42 (pay-per-token API costs, not subscriptions)
SWE-Atlas-QnA-AA rewords 127 of 1,306 rubric criteria and uses GPT-6 Luna (xhigh) as judge with five votes per criterion; dataset and judge config are on Hugging Face
Independent AI evaluation platform Artificial Analysis (@ArtificialAnlys) officially benchmarked DeepSeek's web search API (DeepSeek Search). Augmenting the base model with DeepSeek Search boosted its Search Index score from 33 to 69—an impressive +36 point leap (ranking 17th among 27 tested search APIs). DeepSeek Search established a market-leading price-performance ratio, priced at just $0.75 per 1,000 searches (~$0.032 per completed task), making it the most cost-effective paid search API evaluated. It recorded 70% accuracy on AA-Omniscience and 63% on BrowseComp, highlighting an optimal balance between factual grounding and operational economics.
Zoom
Key Takeaways
Artificial Analysis independently benchmarked DeepSeek Search API, lifting Search Index score to 69 (+36 pts over base model)
Industry-leading cost efficiency: priced at $0.75 per 1,000 searches (~$0.032 per task), the cheapest paid search API on the market
Scores 70% accuracy on cross-domain factual QA benchmark AA-Omniscience (ranking 10th out of 27 configurations)
Achieves 63% accuracy on long-horizon web navigation benchmark BrowseComp, offering high-ROI grounding for agent developers
Artificial Analysis launched AA-PersonalityBench (@ArtificialAnlys), an innovative behavioral benchmark quantifying psychological traits in frontier LLMs, with initial rankings focusing on Confidence (uncertainty calibration) and Sycophancy (tendency to agree with user misconceptions). The data revealed that Google's Gemini 3.8 Flash (100) and Gemini 4 Argon (97) displayed the highest assertiveness while being the least likely to admit uncertainty. Conversely, on sycophancy tests, Llama 4 Maverick scored 97 (frequently validating flawed user prompts), whereas Claude Fable 5.1 scored a near-flawless 4, refusing to flatter and rigorously holding factual ground.
Zoom
Key Takeaways
Artificial Analysis rolls out AA-PersonalityBench, evaluating behavioral alignment across Confidence and Sycophancy
Gemini 3.8 Flash (100) and Gemini 4 Argon (97) top confidence scores, yet exhibit minimal willingness to express epistemic uncertainty
Llama 4 Maverick recorded a high sycophancy score of 97, regularly echoing invalid user premises
Claude Fable 5.1 (4) and Claude Opus 5.5 (46) demonstrated elite resistance to user gaslighting, strictly preserving factual neutrality
Independent AI evaluation platform Artificial Analysis officially released HealthBench Professional alongside the length-normalized MLCR-AA index (@ArtificialAnlys). The benchmark revealed a pervasive industry blindspot: LLM-as-a-Judge evaluations suffer from acute 'verbosity bias', disproportionately rewarding longer generations regardless of medical conciseness. In raw HealthBench scoring where Anthropic outputs were 4–6x longer, Claude Opus 5.5 led at 71.4%. However, once length bias was scientifically normalized via MLCR-AA, GPT-6 Astra (max) tied Claude Sonnet 5.5 for #1, while Claude Opus 5.5 plummeted from #1 down to #6, setting a new rigorous standard for medical model evaluation.
Zoom
Key Takeaways
Artificial Analysis unveils HealthBench Professional and the length-calibrated MLCR-AA benchmark
On Oct 21, Baseten's research team Base Labs released open decision models mammoth-1 (27B) and elephant-1 (9B), full fine-tunes of Qwen3.5 that skip text generation and return per-option probabilities in one forward pass for routing, classification and scoring. Per Baseten, mammoth-1 tops the Jev Decision Index 0.4 at 71.24 and elephant-1 ties for #1 on the Artificial Analysis Decision Index. Weights are on Hugging Face and the Baseten API is free through Oct 27.
Zoom
Key Takeaways
Benchmarks (vendor): mammoth-1 scores 71.24 on Jev Decision Index 0.4, #1 of 208 decision models; 65.9 on AA Decision Index
elephant-1 (9B): 62.6 on Jev, best 9B; 65.2 on AA Decision Index, tied #1 with d1-ultra
Training: full fine-tunes of Qwen3.5-27B / 9B on 6.21M decision examples incl. abstention and safety data
Mechanism: block-bidirectional attention plus shared-prefix caching, one forward pass for many questions; up to ~10x faster than a reasoning model (vendor claim)
Try it: Hugging Face weights (MIT); Baseten API free until Oct 27 11:59 PM PT with zero data retention
Artificial Analysis released an independent evaluation of Cognition's new coding agent, Devin Fusion. Moving beyond single-model execution, Devin Fusion employs an asymmetric multi-agent architecture pairing Claude Opus 5.5 (max) as the lead reasoning orchestrator alongside GPT-6 Luna (high) as a high-speed parallel worker. In the standardized Coding Agent Index, Devin Fusion achieved an elite score of 68.1 (matching Muse Code at 68.0 and rivaling Claude Code Sonnet 5.5 at 68.4). Crucially, the hybrid architecture slashes average task expense from $13.04 down to $9.01 (a 31% cost reduction) while shortening execution runtime from 65 to 55 minutes.
Zoom
Key Takeaways
Artificial Analysis independently benchmarked Devin Fusion, scoring 68.1 on the standardized Coding Agent Index
Pioneers asymmetric model collaboration: Claude Opus 5.5 orchestrates architecture while GPT-6 Luna executes parallel sub-tasks
Cuts per-task execution costs from $13.04 to $9.01 (a 31% savings) and trims average completion time from 65 to 55 minutes
Outperforms Claude Code with Opus 5.5 on DeepSWE v1.1 (72.6% vs 68.4%) and SWE-Atlas-QnA (69.6% vs 66.4%)
Mistral AI unveiled Voxtral 3, a production audio intelligence suite paired with a real-time Voice-Agent API. The lineup features the 4B-parameter Mini Transcribe, Small Transcribe 3, and a turn-key voice agent SDK built on LiveKit. Supporting transcription across 50 languages with diarization, multi-lingual TTS with emotional tags across 35+ languages, and 30-minute voice cloning, Voxtral 3 redefines voice AI economics. In Artificial Analysis independent benchmarks, Small Transcribe 3 secured rank #3 globally with a 2.1% error index (outperforming ElevenLabs Scribe v2), while Realtime Transcribe delivers sub-500ms latency at $0.04/min for full voice agent turns.
Zoom
Key Takeaways
Mistral AI launched the Voxtral 3 suite spanning transcription, emotional TTS, studio-grade voice cloning, and a LiveKit-powered Voice-Agent SDK
Artificial Analysis independently verified Small Transcribe 3 at a 2.1% error index, outperforming ElevenLabs Scribe v2 (2.2%) and Grok Voice 2.0 (2.3%)
On Oct 20, Krea said Krea 3, the image-editing model it released on Oct 15, has entered several third-party leaderboards. On Arena's Multi-Image Edit board, Krea 3 Ultra ranks #4 at 1435±11 (preliminary, 2,991 votes), behind three OpenAI models; it is #8 on single-image editing. On Artificial Analysis, Ultra scores 1151 Elo (#5) and Fast scores 1125 (#11). Krea 3 edits at native 2K, returns most edits in under 8 seconds and costs $0.06 per image.
Zoom
Key Takeaways
Arena Multi-Image Edit: Krea 3 Ultra is #4 at 1435±11 on 2,991 votes, marked preliminary; the top 3 are two GPT-Image-2.5 variants and GPT-Image-2
Arena single-image Image Edit: Krea 3 Ultra is #8 at 1411±11 on 2,619 votes
Artificial Analysis Image Editing v2.1: Ultra is #5 at 1151 Elo (statistically tied for #3 with Nano Banana 2.1 and MAI-Image-2.6); Fast is #11 at 1125
Design Arena Image Editing: #4 of 51 models at 1298 Elo
Vendor figures: most edits under 8 seconds, native 2K, $0.06 per image, up to 6 reference images
