The error the headline metric cannot see
Word error rate answers one question: did the transcript get the words right? It says nothing about who said them. A meeting summary built on a transcript that attributes a commitment to the wrong person is not mostly right. It is wrong in the way that matters, and until now Artificial Analysis had no number for it.
The Diarization Index is that number. Artificial Analysis calls each API with diarization on, gives it no prompt and does not tell it how many people are in the room. Every scored reference word is then checked: right speaker, wrong speaker, or missed, where missed means the model had nobody speaking while the word was said. Add the wrong and the missed, divide by the scored words, and that is WDER. The headline index averages three bands equally, two speakers, three to four, and five to eight, so an easy two-person call cannot hide a crowded meeting.
Two scoring choices deserve attention. Fillers such as "uh" are not scored, but backchannels such as "mm-hmm" are, which is correct: in a meeting, who agreed is information. And a band score only appears when a model returns a valid, speaker-labeled result for every conversation in that band. A band is published only when every conversation in it has a scored, speaker-labeled result.
What the first table says
Rankings, band results, speeds and prices below describe AA's November 13 launch table, as reported by AlphaSignal; the live board may differ. At the index's launch, Nemotron 3.5 Transcribe leads at 3.3%, less than half the 7.5% of the next entry, Voxtral Small Transcribe 3. That gap widens in the five-to-eight-speaker band. At two speakers Nemotron is at 1.7%; at five to eight it rises to 5.6%, while every other API in the index passes 10%. One word in ten, in a crowded room, is put in the wrong mouth or dropped by the best of the rest.
The trade-offs are visible in the same table. Microsoft's MAI-Transcribe-2 is third on accuracy at 8.8% but runs at a 313.9x speed factor (audio duration divided by response time, including the wait for asynchronous jobs to finish) for $1.67 per 1,000 minutes, against Voxtral's 60.1x at $4.00. Nemotron is listed as open weights with no standard per-minute price, and the score on the board belongs to a DeepInfra endpoint, not to every possible self-hosted deployment. Hardware, inference stack and operating cost are the buyer's problem, and the number may move with them.
Context from the same leaderboard keeps the result in proportion. On Artificial Analysis' word error index, Nemotron sits at 2.8%, level with Voxtral Small and behind Salesforce's SOV-Transcribe-1 at 2.3%. It is not the runaway leader at getting words down. On this test it records the lowest error at saying who said them. Those are different claims, and a word error ranking cannot answer the second one.
Where the test stops
The limits are stated in the methodology and should travel with the headline. The set is English only. Most of it is held out. Holding material private reduces exposure but does not establish freedom from contamination, and nobody outside Artificial Analysis can rerun the test exactly. Results carry 95% confidence intervals. Read close point estimates alongside their confidence intervals; the intervals alone do not establish whether models differ reliably. And the index covers hosted APIs that return speaker labels in batch mode; it says nothing about live transcription, where attribution has to happen before a speaker has finished.
None of that undermines the exercise. It defines it. For two-person calls, compare the two-speaker band alongside measured speed, price and your own audio. For an eight-person meeting whose notes will be quoted back at people, the band breakdown, not the headline average, is the figure to shop on.
Sources
- [1] Artificial Analysis, Speech to Text (ASR) Providers Leaderboard — Diarization Index section (index definition; read November 14, 2026)Read source
- [2] Artificial Analysis, Speech to Text Benchmarking Methodology — Diarization Evaluation (WDER formula, dataset, conditions, band weighting; read November 14, 2026)Read source
- [3] AlphaSignal, "Nvidia's Nemotron 3.5 Beats Every API at Identifying Who Said What" (November 13, 2026; top-three table, band detail and DeepInfra note, reporting Artificial Analysis data)Read source
- [4] The Daily Fledgling, Morning Edition (November 14, 2026; launch corroboration in its speech and images roundup)Read source