On November 13 Artificial Analysis published the Diarization Index, a benchmark for whether speech to text APIs attribute each word to the person who actually said it. It scores Word Diarization Error Rate (WDER), the share of words given to the wrong speaker or to no speaker, across 97 English conversations of two to eight people, averaging three speaker bands (2, 3-4 and 5-8) with equal weight. NVIDIA's open-weight Nemotron 3.5 Transcribe leads at 3.3%, ahead of Voxtral Small Transcribe 3 at 7.5%, the most accurate priced API; MAI-Transcribe-2 scores 8.8% and is one of the fastest.
Key Takeaways
- ✓The new metric is WDER = (wrong-speaker words + missed words) / scored reference words, lower is better; AA says it ranks on WDER instead of DER because for APIs DER mostly scores word timestamps rather than who spoke
- ✓The set has 97 conversations of about 2-9 minutes each, about 6.8 hours in total, mixing real and synthetic speech; most audio is proprietary and held out, plus about 126 minutes of public NOTSOFAR-1 meeting recordings
- ✓APIs are called directly with diarization switched on and are not told how many people are talking, with each conversation sent whole; price includes any separate diarization charge
- ✓Nemotron 3.5 Transcribe goes from 1.7% WDER with two speakers to 5.6% with five to eight; in the largest groups every other API gets more than 10% of words wrong
- ✓Voxtral Small Transcribe 3 scores 7.5% at $4.00 per 1,000 minutes; MAI-Transcribe-2 scores 8.8% at $1.67 and transcribes 93.7 seconds of audio per second of waiting; Nemotron 3.5 Transcribe is open-weight and has no listed price
Key Decision Metrics at a Glance
Turn your technical choice into a development budget
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Background
Meeting notes, call QA, podcast captions and voice agents all depend on diarization: even a perfect transcript breaks summaries and automations if words land on the wrong person. Most leaderboards compare only WER. On November 13 Artificial Analysis turned "who said what" into its own Diarization Index.
Architecture highlights
WDER checks every scored word: given to the person who said it is right, given to someone else is a wrong-speaker error, and no one talking at that moment is a miss. The set has 97 English conversations of two to eight people, about 6.8 hours, split into 2, 3-4 and 5-8 speaker bands; the index is the equal mean of the three band WDERs, with 95% confidence intervals. APIs run with diarization on, no speaker count hint, and each conversation sent whole.
Benchmarks and data
Per AA's posts: Nemotron 3.5 Transcribe leads at 3.3% (1.7% with two speakers, 5.6% with five to eight); Voxtral Small Transcribe 3 is second at 7.5% for $4.00 per 1,000 minutes; MAI-Transcribe-2 scores 8.8% for $1.67 with a 93.7x speed factor. In the largest groups every API except Nemotron exceeds 10%. The full ranking is on AA's leaderboard.
Developer quick start
For multi-person meetings, look at the 5-8 speaker band rather than the headline score; on a budget, weigh MAI-Transcribe-2 against Voxtral. AA notes this version covers hosted speech to text APIs only, not self-hosted or third-party-hosted models. Re-test on a small sample of your own recordings before choosing.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.