Google DeepMind and Artificial Analysis Launch MKA, a Multimodal Knowledge Agent Benchmark of 200 Real Visual Tasks: Claude Fable 5.2 Leads at 57%, Gemini 4 Carbon Scores 54% at About One-Fifth the Cost per Task

Core Background and Industry Pain Points

Most knowledge work benchmarks can be solved with text alone, so models rarely need to look at an image or a page layout. Real work is far more visual: architects cross-check plans against elevations, engineers trace schematics, clinicians read scans, and the results end up in decks or spreadsheets. Google DeepMind says existing evaluations do not cover this, so it built MKA with Artificial Analysis and released it on November 10 alongside Gemini 4 Carbon.

Architecture Highlights and Underlying Mechanisms

DeepMind worked with experts across 10 verticals and 83 occupations to write 200 tasks. Each pairs a bundle of multimodal input files with about 70 expert rubric checks and represents 3 to 8 hours of work for a seasoned professional. Checks are split into Regression (baseline) and Climb (harder, separating strong models), and the headline score is Climb WSR, the weighted satisfaction rate. Every model runs in Artificial Analysis's open-source Stirrup harness with code execution, web search, and image viewing for up to 250 turns, and a non-Google model, GPT-6 Luna, grades the deliverables, agreeing with experts 96% of the time. The 200 tasks stay private to avoid training-data leaks, and 10 public tasks are released separately.

Authoritative Benchmarks and Measured Comparisons

Artificial Analysis tested 21 models at launch: Claude Fable 5.2 (max) scores 57%, Gemini 4 Carbon (high) 54%, and Claude Opus 5.5 (max) 53%. Costs differ sharply: Fable averages $31.97 per task, Carbon $6.29, and GPT-6.1 Sol $1.11. No model wins every vertical, and Carbon leads only Healthcare; Word and plain-text outputs are hardest, with no model above 52%. In one example task, Fable took 224 turns, viewed images 203 times, and spent about $68. Note that Google co-designed MKA, so the claim that Carbon offers the best value is Google's own framing.

Developer Adoption and Getting Started

To evaluate your own agent, start with the 10 public tasks on Hugging Face to see the input files and rubric format, then run them in the open-source Stirrup harness with the same tools and turn limit. When choosing a model, look past the headline score: teams doing architecture or engineering reviews should check per-vertical results, and cost-sensitive teams can use Artificial Analysis's score-versus-cost chart to pick a model on the Pareto frontier.