Artificial Analysis launches Diarization Index, NVIDIA leads
A standard WDER metric shifts how transcription APIs will be evaluated, since many buyers currently judge quality only on word accuracy and miss attribution failures that break meeting summaries and call analytics. Teams building voice agents, meeting assistants or call-centre tooling can use the 97-conversation suite to benchmark vendors before committing. NVIDIA's Nemotron 3.5 Transcribe at 3.3% versus Voxtral Small Transcribe 3 at 7.5% sets a clear quality bar for providers to chase.
Artificial Analysis has announced the Diarization Index, a new benchmark measuring how well speech-to-text APIs attribute transcribed words to the correct speaker. The index scores APIs on Word Diarization Error Rate (WDER), defined as the share of words assigned to the wrong speaker or to no speaker at all.
The benchmark is built on 97 conversations ranging from two to eight participants, with equal weighting given to each group size. This design tests whether providers can handle both simple two-person calls and complex multi-party meetings, rather than being skewed toward easy cases.
On the first results, NVIDIA's Nemotron 3.5 Transcribe leads at 3.3% WDER, well ahead of Voxtral Small Transcribe 3 at 7.5%. Speaker attribution errors remain one of the largest quality gaps in production transcription pipelines, so a standardised public metric gives buyers a consistent way to compare transcription APIs beyond raw word error rate.
Announcing the Artificial Analysis Diarization Index, our new benchmark for who said what in speech to text
The Diarization Index scores APIs on Word Diarization Error Rate (WDER): the share of words given to the wrong speaker or to no speaker, across 97 conversations of two to eight people, with equal weight on each group size.
Nemotron 3.5 Transcribe from @NVIDIAAI leads at 3.3%, ahead of Voxtral Small Transcribe 3 at 7.5%. Thread below 🧵
This report is categorized under AITOP's "Products" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include Nemotron, Voxtral, NVIDIA, speech recognition.
Source attribution: curated from Artificial Analysis (@ArtificialAnlys).
