On the CALLHOME telephone benchmark, the same end-to-end diarization system that scores 8.50% DER on two-speaker calls scores 40.29% on six-speaker calls (Horiguchi et al., EEND-EDA, Interspeech 2020). Speaker diarization is the “who spoke when” step: it splits a recording into stretches and labels each with a speaker, before or alongside transcription. On curated benchmarks it has improved quickly, with the median DIHARD Track 1 error falling from over 30% in DIHARD I to under 20% in DIHARD III (Ryant et al., Third DIHARD Challenge 2021), but dinner-party audio still produced a 62.0% baseline DER in CHiME-6 (Watanabe et al., CHiME-6 Challenge overview 2020). We aggregated data from the DIHARD and VoxSRC challenge organizers, the CHiME-6, NOTSOFAR-1 and DISPLACE challenges, ETH Zurich’s 2025 diarization benchmark, Brno University of Technology, the pyannote project’s model cards, and peer-reviewed clinical studies. For the word-level side of the same pipeline, see our roundup on speech recognition accent bias and word error rate.
A note on reading every number below: DER (diarization error rate) is lower-is-better, and it is only comparable when two results share the same benchmark, the same “collar” (a short no-score window around each speaker change) and the same rule on overlapped speech. Each table states the setup when the source does.
TL;DR
- CALLHOME DER climbs from 8.50% with 2 speakers to 40.29% with 6 speakers for the same system (Horiguchi et al., Interspeech 2020).
- Median DIHARD Track 1 DER fell from over 30% (DIHARD I) to 25% (DIHARD II) to under 20% (DIHARD III) (Ryant et al., Third DIHARD Challenge 2021).
- VoxSRC winners scored 3.71% DER in 2020 and 4.30% in 2023 on YouTube audio, roughly half the baseline each year (Huh et al., VoxSRC retrospective, IEEE TASLP 2024).
- Removing the 0.25 s collar raises AMI DER from 9.8% to 15.4% for the same system (Han et al., Brno University of Technology 2024).
- Over 33% of errors on single-microphone NIST meeting data fall within 0.5 s of a speaker change (Knox, UC Berkeley 2013).
- An average of 12% to 15% of speech in meeting recordings is overlapped (Park et al., Computer Speech and Language 2022).
- Doctor-patient dialogues averaged 33.38% DER versus 15.18% for general talk across eight systems (Okocha et al., 2025).
- The CHiME-6 dinner-party baseline scored 62.0% DER on the evaluation set (Watanabe et al., CHiME-6 overview 2020).
- The best NOTSOFAR-1 meeting system cut the speaker-attributed word error rate to 10.8% from a 28.3% baseline (Abramovski et al., NOTSOFAR-1 summary 2025).
- The fastest model in a 2025 benchmark processed audio 214.3 times faster than real time (Lanzendoerfer et al., ETH Zurich 2025).
1. What DER Measures, and Why the Scoring Setup Moves the Number
DER adds up three kinds of time and divides by the total amount of reference speech: missed speech (someone talked, the system heard silence), false alarm (the system marked speech where there was none) and speaker confusion (speech was detected but given to the wrong person). The pyannote 3.1 model card shows the arithmetic plainly: on AMI headset audio its 18.8% DER is 3.6% false alarm plus 9.5% miss plus 5.7% confusion.
The bigger trap is the collar. Since the NIST Rich Transcription evaluations, most papers ignore 0.25 s on each side of every reference boundary, because human annotators rarely agree on the exact millisecond a turn starts. That single choice removes 5.6 points, or 36% of the no-collar figure: the same Brno University of Technology system scores 15.4% on AMI without a collar and 9.8% with one (Han et al., 2024). The forgiven zone is exactly where systems fail most, since a third or more of all errors sit within half a second of a speaker change (Knox, UC Berkeley technical report 2013).
| Metric | Value | Source |
|---|---|---|
| Standard no-score collar around each reference segment boundary (RT-style scoring) | 0.25 s, described as the most widely used scheme | Park et al., Computer Speech and Language 2022 |
| DIHARD and CHiME-6 scoring rule | No collar, overlapped speech scored | Park et al., Computer Speech and Language 2022 |
| AMI DER, BUT WavLM-based system, no collar vs 0.25 s collar | 15.4% vs 9.8% | Han et al., Brno University of Technology 2024 |
| AliMeeting DER, same system, no collar vs 0.25 s collar | 17.6% vs 10.2% | Han et al., Brno University of Technology 2024 |
| AMI DER, pyannote 3 baseline, no collar vs 0.25 s collar | 21.1% vs 13.7% | Han et al., Brno University of Technology 2024 |
| Share of errors within 0.5 s of a speaker change, NIST RT-09 meetings, single vs multiple distant microphones | Over 33% vs 40% | Knox, UC Berkeley 2013 |
| AMI headset-mix DER, pyannote 3.1, split into false alarm + miss + confusion | 18.8% = 3.6% + 9.5% + 5.7% (project-reported) | pyannote, speaker-diarization-3.1 model card |
Context: the Berkeley error analysis is the most recent available data on error position (Knox, UC Berkeley 2013). The 2025 ETH benchmark points the same way: missed speech was the dominant error type for most models, and the authors attribute missed segments, which average about 350 ms, to onset and offset timing rather than to whole utterances going unheard (Lanzendoerfer et al., Benchmarking Diarization Models 2025).
2. State of the Art by Benchmark, and How Fast DER Has Fallen
The progress is real, but it is benchmark-specific. In DIHARD, a deliberately hard mix of 11 domains, median Track 1 DER fell from over 30% to under 20% across three editions (Ryant et al., The Third DIHARD Diarization Challenge 2021). Track 1 hands systems the true speech regions; Track 2 makes them find speech on their own, which is why its numbers run higher.
On broadcast-style YouTube audio, VoxSRC winners sit around 4%, about half the organizers’ baseline each year (Huh et al., The VoxCeleb Speaker Recognition Challenge: A Retrospective, IEEE TASLP 2024). The 2020 and 2023 winners are not a time series, though: the test set grew from 232 files to 413 files, so a slightly higher 2023 figure does not mean the field went backward.
| Metric | Value | Source |
|---|---|---|
| Median DIHARD Track 1 DER (true speech regions given), DIHARD I vs II vs III | Over 30% vs 25% vs under 20% | Ryant et al., Third DIHARD Challenge 2021 |
| Median DIHARD Track 2 DER (system detects speech itself), DIHARD I vs III | 40% vs under 25% | Ryant et al., Third DIHARD Challenge 2021 |
| DIHARD III winner (USTC-NELSLIP), core evaluation set, Track 1 vs Track 2 | 13.45% vs 19.37% (no collar, overlap scored) | DIHARD III challenge results |
| VoxSRC diarization track, 2020 test set, winner vs baseline | 3.71% vs 8.15% (0.25 s collar, overlap scored) | Huh et al., VoxSRC retrospective 2024 |
| VoxSRC diarization track, 2023 test set, winner vs baseline | 4.30% vs 8.54% (0.25 s collar, overlap scored) | Huh et al., VoxSRC retrospective 2024 |
| CALLHOME DER, all calls, end-to-end EEND-EDA vs x-vector clustering | 15.29% vs 19.43% (0.25 s collar) | Horiguchi et al., Interspeech 2020 |
| Average DER of the ICSI system on NIST RT-09 meetings, single distant microphone | 31.3% | Sinclair and King, University of Edinburgh 2013 |
Context: the DIHARD III winner’s DER on the full evaluation set was 11.3% on Track 1 and 16.78% on Track 2, against a baseline of 20.65% and 27.34% on the core set (DIHARD III challenge results). The RT-09 figure is the most recent available data for that evaluation (Sinclair and King, Where are the challenges in speaker diarization? 2013), and it comes from different data and a different era than DIHARD, so it should not be read as a direct before-and-after comparison.
3. Overlapping Speech and Speaker Count: The Two Biggest DER Drivers
Overlap is not a corner case. An average of 12% to 15% of speech in meeting recordings is overlapped, and the review that collected this figure notes it can rise further in everyday conversation (Park et al., A Review of Speaker Diarization, Computer Speech and Language 2022). Older clustering pipelines give each moment exactly one speaker, so in a two-person overlap at least half the scored time is wrong before any other mistake happens.
Speaker count compounds the problem. On CALLHOME, the same EEND-EDA model goes from 8.50% DER on two-speaker calls to 40.29% on six-speaker calls (Horiguchi et al., EEND-EDA, Interspeech 2020), and models with a hard speaker limit pay twice: the Sortformer family, capped at four speakers, showed more speaker confusion on crowded recordings in the ETH benchmark. Multi-microphone capture helps, as the NOTSOFAR-1 meeting results in section 5 show, a pattern also visible in our data on far-field beamforming and wake-word accuracy.
| Metric | Value | Source |
|---|---|---|
| Average share of overlapped speech in meeting recordings | 12% to 15% | Park et al., Computer Speech and Language 2022 |
| Average overlap share: AliMeeting vs AMI vs CALLHOME vs VoxConverse | 19% vs 15.94% vs 12.58% vs 3.52% | Lanzendoerfer et al., ETH Zurich 2025 |
| CALLHOME DER, EEND-EDA, calls with 2 vs 3 vs 4 speakers | 8.50% vs 13.24% vs 21.46% | Horiguchi et al., Interspeech 2020 |
| CALLHOME DER, EEND-EDA, calls with 5 vs 6 speakers | 33.16% vs 40.29% | Horiguchi et al., Interspeech 2020 |
| CALLHOME DER, x-vector clustering, calls with 2 vs 6 speakers | 15.45% vs 34.27% | Horiguchi et al., Interspeech 2020 |
| DER on 4-speaker recordings: Sortformer (4-speaker cap) vs DiariZen | 21.3% vs 12.7% (0.25 s collar, overlap scored) | Lanzendoerfer et al., ETH Zurich 2025 |
| Average speakers per recording, VoxSRC 2020 test set | 6.5 speakers over an average 675.6 s | Huh et al., VoxSRC retrospective 2024 |
Context: speaker count alone does not decide the score. In the ETH benchmark the best system scored 6.6% on recordings with 5 or more speakers but 10.1% on 4-speaker ones, a reminder that overlap and recording conditions matter as much as head count. The CALLHOME 5- and 6-speaker results also rest on few files: the EEND-EDA authors note CALLHOME has only ten recordings with more than four speakers.
4. Domain Gaps: Clinical, Child, Multilingual and In-the-Wild Audio
A model tuned on meetings and phone calls does not carry its accuracy into a clinic, a family home or online video. Across eight diarization systems, doctor-patient dialogues averaged 33.38% DER against 15.18% for general conversations, an 18.20 percentage-point gap (paired t-test p < 10⁻⁴), and false alarms caused most of the clinical penalty (Okocha et al., Domain-Aware Speaker Diarization on African-Accented English 2025). Fine-tuning on accented speech halved one open model’s error, but the clinical gap shrank rather than disappeared.
Children are a separate frontier. In the BabyHuBERT study, child-centered daylong recordings are scored per speaker type (key child, other children, adult men, adult women) with an F1 score, where higher is better, and the best 2026 model still sits below a second human annotator (Charlot et al., BabyHuBERT 2026). DIHARD III flagged the same weakness, singling out children in clinical interviews as a particular challenge.
| Metric | Value | Source |
|---|---|---|
| DIHARD III median Track 1 DER, meeting, web video and restaurant domains | 35% to 45% | Ryant et al., Third DIHARD Challenge 2021 |
| DIHARD III DER, clinical interviews and telephone speech, Track 1 vs Track 2 | Generally 10% to 20% vs 15% to 25% | Ryant et al., Third DIHARD Challenge 2021 |
| Average DER across 8 systems, doctor-patient vs general African-accented English dialogues | 33.38% vs 15.18% (no collar, overlap scored) | Okocha et al., 2025 |
| pyannote DER before vs after fine-tuning on African-accented speech, medical dialogues | 31.46% vs 15.73% | Okocha et al., 2025 |
| Child-centered recordings, average F1 across speaker types: LENA vs BabyHuBERT-VTC vs second human annotator | 40.8% vs 65.1% vs 69.8% (higher is better) | Charlot et al., BabyHuBERT 2026 |
| DER on the Ego4D (dev) and AVA-AVD in-the-wild video benchmarks, pyannote community-1 | 46.8% and 44.6% (project-reported, no collar) | pyannote, community-1 model card 2025 |
| DER by language across the ETH benchmark datasets, best system (pyannoteAI), Spanish vs English | 14.3% vs 6.6% | Lanzendoerfer et al., ETH Zurich 2025 |
| Multilingual Indian conversations (DISPLACE), best team DER on the same evaluation data, 2023 vs 2024 | 27.8% vs 21.27% | Kalluri et al., Second DISPLACE Challenge 2024 |
Context: the DISPLACE data covers 9 Indian languages plus Indian English with 3 to 5 participants per session (Kalluri et al., The Second DISPLACE Challenge 2024). The ETH authors link the language gap to data scarcity for some languages and name Spanish as the hardest of the five they tested.
5. Real-World Deployments: Meetings, Call Centers and Clinics
In production, diarization errors surface as transcript lines credited to the wrong person, which is why meeting and medical studies often report word-level metrics. Two appear below: tcpWER, a word error rate that also counts words attributed to the wrong speaker within a time window, and WDER, the share of words given the wrong speaker label. One of the hardest settings ever benchmarked is the dinner party: the CHiME-6 baseline scored 62.0% DER on its evaluation set, and the organizers attribute around 15% absolute of the word error rate gap between their tracks mainly to diarization (Watanabe et al., CHiME-6 Challenge overview 2020).
Office meetings are more tractable. In NOTSOFAR-1, built from 315 real meetings with 4 to 8 participants each in 30 environments, the best multi-channel system reached 10.8% tcpWER (Abramovski et al., Summary of the NOTSOFAR-1 Challenge, Computer Speech and Language 2025). Legal transcription has its own speaker-labeling stakes, covered in our courtroom speech transcription accuracy statistics, and the text errors that sit on top of speaker errors are covered in our speech recognition hallucination data.
| Metric | Value | Source |
|---|---|---|
| CHiME-6 dinner-party baseline DER, development vs evaluation set | 61.6% vs 62.0% | Watanabe et al., CHiME-6 overview 2020 |
| CHiME-6 word error rate gap between tracks attributed mainly to diarization | Around 15% absolute | Watanabe et al., CHiME-6 overview 2020 |
| NOTSOFAR-1 meeting transcription, best system vs baseline tcpWER, multi-channel | 10.8% vs 28.3% | Abramovski et al., NOTSOFAR-1 summary 2025 |
| NOTSOFAR-1 meeting transcription, best system vs baseline tcpWER, single-channel | 22.2% vs 41.4% | Abramovski et al., NOTSOFAR-1 summary 2025 |
| CALLHOME telephone DER, open community-1 vs commercial precision-2 | 26.7% vs 16.6% (vendor-reported, no collar) | pyannote, community-1 model card 2025 |
| WDER across four commercial speech models, 36 primary-care conversations | 1.8% to 13.9% | Tran et al., AMIA Annual Symposium 2022 |
| WDER on physician-patient conversations, conventional pipeline vs joint recognition and diarization model | 15.8% vs 2.2% | El Shafey et al., Interspeech 2019 |
Context: the primary-care recordings were re-enacted by two native English speakers in a quiet studio, so the authors describe their figures as a likely upper bound on real clinic performance (Tran et al., AMIA Annual Symposium Proceedings 2022). The joint model result is most recent available data from a corpus that is not public (El Shafey et al., Interspeech 2019). The precision-2 figure is the vendor’s own claim for its commercial model.
6. Speed, Open Models and Research Scale
Accuracy is only half of a deployment decision. The fastest system in the ETH benchmark ran 214.3 times faster than real time, more than ten times the speed of the open DiariZen model, which ran at 20.2 times real time and scored 13.3% DER overall (Lanzendoerfer et al., ETH Zurich 2025). That trade-off weighs far more on live captioning than on an overnight batch job.
Open models are widely used. The pyannote pipelines show millions of monthly downloads, and the project’s own September 2025 benchmark reports lower DER for its current open release than for its legacy one on most datasets, with its commercial tier ahead of both (pyannote, speaker-diarization-community-1 model card). Treat those scores as project-reported: they are run by the same team that ships the models.
| Metric | Value | Source |
|---|---|---|
| Real-time factor (audio length divided by processing time), Sortformer v2 vs Sortformer v2 streaming | 214.3x vs 209.5x | Lanzendoerfer et al., ETH Zurich 2025 |
| Real-time factor, open pyannote vs DiariZen | 45.0x vs 20.2x | Lanzendoerfer et al., ETH Zurich 2025 |
| Overall DER across the ETH benchmark, pyannoteAI vs DiariZen | 11.2% vs 13.3% (0.25 s collar, overlap scored) | Lanzendoerfer et al., ETH Zurich 2025 |
| DIHARD 3 full DER, pyannote 3.1 vs community-1 vs precision-2 | 21.4% vs 20.2% vs 14.7% (project and vendor-reported, no collar) | pyannote, community-1 model card 2025 |
| Monthly downloads on Hugging Face, pyannote speaker-diarization-3.1 vs community-1 | 6,886,119 vs 5,051,937 (viewed October 2026) | Hugging Face model pages |
| NOTSOFAR-1 simulated training data released to participants | About 1,000 hours | Abramovski et al., NOTSOFAR-1 summary 2025 |
| AMI Meeting Corpus size | 100 hours, about two-thirds role-played design meetings | University of Edinburgh, AMI Meeting Corpus |
Context: the AMI Meeting Corpus, still a core diarization benchmark, records close-talking and far-field microphones in sync (University of Edinburgh, AMI Meeting Corpus). The ETH benchmark could not measure pyannoteAI’s speed because only API access was available.
Summary: Speaker Diarization Error Rate by the Numbers
| Metric | Value | Source |
|---|---|---|
| CALLHOME DER, EEND-EDA, 2 vs 6 speakers (0.25 s collar) | 8.50% vs 40.29% | Horiguchi et al., Interspeech 2020 |
| CALLHOME DER, EEND-EDA vs x-vector clustering, all calls | 15.29% vs 19.43% | Horiguchi et al., Interspeech 2020 |
| Median DIHARD Track 1 DER, DIHARD I vs III | Over 30% vs under 20% | Ryant et al., Third DIHARD Challenge 2021 |
| DIHARD III winner, core evaluation set, Track 1 vs Track 2 | 13.45% vs 19.37% | DIHARD III challenge results |
| VoxSRC winner vs baseline, 2023 test set | 4.30% vs 8.54% | Huh et al., VoxSRC retrospective 2024 |
| AMI DER, same system, no collar vs 0.25 s collar | 15.4% vs 9.8% | Han et al., Brno University of Technology 2024 |
| Errors within 0.5 s of a speaker change, NIST RT-09, single distant microphone | Over 33% | Knox, UC Berkeley 2013 |
| ICSI system DER, NIST RT-09 single distant microphone | 31.3% | Sinclair and King, University of Edinburgh 2013 |
| Overlapped share of speech in meeting recordings | 12% to 15% | Park et al., Computer Speech and Language 2022 |
| Overlap share, AliMeeting vs AMI | 19% vs 15.94% | Lanzendoerfer et al., ETH Zurich 2025 |
| Doctor-patient vs general dialogue DER, 8-system average | 33.38% vs 15.18% | Okocha et al., 2025 |
| Child-centered recordings, average F1, best model vs second human annotator | 65.1% vs 69.8% | Charlot et al., BabyHuBERT 2026 |
| DISPLACE best team DER, 2023 vs 2024 | 27.8% vs 21.27% | Kalluri et al., Second DISPLACE Challenge 2024 |
| CHiME-6 dinner-party baseline DER, evaluation set | 62.0% | Watanabe et al., CHiME-6 overview 2020 |
| NOTSOFAR-1 best vs baseline tcpWER, multi-channel | 10.8% vs 28.3% | Abramovski et al., NOTSOFAR-1 summary 2025 |
| WDER range, four commercial speech models, primary care | 1.8% to 13.9% | Tran et al., AMIA Annual Symposium 2022 |
| pyannote 3.1 monthly downloads on Hugging Face | 6,886,119 | Hugging Face, pyannote speaker-diarization-3.1 model page |
Methodology and Sources
- Ryant et al., The Third DIHARD Diarization Challenge (2021): 11 domains, medians across DIHARD editions, per-domain ranges, scoring without collars and with overlap.
- DIHARD III challenge results: official Track 1 and Track 2 leaderboard, core and full evaluation sets.
- Huh et al., The VoxCeleb Speaker Recognition Challenge: A Retrospective (IEEE TASLP 2024): diarization track winners and baselines, 0.25 s collar with overlap scored.
- Horiguchi et al., End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractors (Interspeech 2020): CALLHOME DER by number of speakers.
- Han et al., Leveraging Self-Supervised Learning for Speaker Diarization (Brno University of Technology, 2024): collar 0 s vs 0.25 s results on AMI, AISHELL-4 and AliMeeting.
- Lanzendoerfer et al., Benchmarking Diarization Models (ETH Zurich, 2025): about 196.6 hours across CALLHOME, VoxConverse, AMI and AliMeeting; overlap shares, per-language and per-speaker-count DER, speed.
- Park et al., A Review of Speaker Diarization: Recent Advances with Deep Learning (Computer Speech and Language, 2022): collar conventions and meeting overlap share.
- Knox, Speaker Diarization: Current Limitations and New Directions (UC Berkeley technical report, 2013): error position relative to speaker changes on NIST RT-09.
- Sinclair and King, Where are the challenges in speaker diarization? (University of Edinburgh, 2013): RT-09 meeting DER.
- Watanabe et al., CHiME-6 Challenge overview (2020): dinner-party baseline DER and the diarization share of the word error gap.
- Abramovski et al., Summary of the NOTSOFAR-1 Challenge (Computer Speech and Language, 2025): meeting dataset and tcpWER results.
- Kalluri et al., The Second DISPLACE Challenge (2024): multilingual conversational diarization results.
- Okocha et al., Domain-Aware Speaker Diarization on African-Accented English (2025): clinical vs general DER across eight systems.
- Charlot et al., BabyHuBERT (2026): speaker segmentation in child-centered long-form recordings.
- Tran et al., Automatic speech recognition performance for digital scribes (AMIA Annual Symposium Proceedings, 2022): WDER in primary-care conversations.
- El Shafey et al., Joint Speech Recognition and Speaker Diarization via Sequence Transduction (Interspeech 2019): WDER on physician-patient conversations.
- pyannote, speaker-diarization-3.1 model card and pyannote, speaker-diarization-community-1 model card: project-reported DER (no collar, overlap scored), download counts; precision-2 figures are the vendor’s own claims for a commercial model.
- University of Edinburgh, AMI Meeting Corpus: corpus size and recording setup.
- Data watch: DER values are only comparable within one benchmark and one scoring setup; VoxSRC uses a 0.25 s collar while DIHARD, the pyannote model cards and the African-accented clinical study use none, so do not compare a VoxSRC 4.30% with a DIHARD 13.45%. The pyannote 3.1 model card lists 21.7% on DIHARD 3 full while the newer community-1 card lists 21.4% for the same legacy pipeline, a small discrepancy from evaluation updates; we use the community-1 table for version comparisons. The VoxSRC retrospective’s 2021 and 2022 rows are internally inconsistent between its table and text, so we report only 2020 and 2023. The African-accented clinical study’s per-component error table does not sum exactly to its stated medical DER, so we quote only the headline averages. The NOTSOFAR-1 summary labels its single-channel and multi-channel best results inconsistently in one table; we follow the text, whose 51% relative improvement matches the figures above. Hugging Face download counts are a rolling monthly snapshot and change daily. The Berkeley, Edinburgh, CHiME-6, DIHARD III, EEND-EDA, Interspeech 2019 and AMIA 2022 figures are the most recent available data for their specific measurements.
- Last updated: October 27, 2026. We update this roundup quarterly, and the next refresh is expected when the Interspeech 2026 challenge summary papers and the next pyannote model-card benchmark update are published.