G-Root/speaker-diarization-optimized
0167
1---2tags:3 - pyannote4 - pyannote-audio5 - pyannote-audio-pipeline6 - audio7 - voice8 - speech9 - speaker10 - speaker-diarization11 - speaker-change-detection12 - voice-activity-detection13 - overlapped-speech-detection14 - automatic-speech-recognition15license: mit16extra_gated_prompt: "The collected information will help acquire a better knowledge of pyannote.audio userbase and help its maintainers improve it further. Though this pipeline uses MIT license and will always remain open-source, we will occasionnally email you about premium pipelines and paid services around pyannote."17extra_gated_fields:18 Company/university: text19 Website: text20---21 22Using this open-source pipeline in production? 23Make the most of it thanks to our [consulting services](https://herve.niderb.fr/consulting.html).24 25# 🎹 Speaker diarization 3.126 27This pipeline is the same as [`pyannote/speaker-diarization-3.0`](https://hf.co/pyannote/speaker-diarization-3.1) except it removes the [problematic](https://github.com/pyannote/pyannote-audio/issues/1537) use of `onnxruntime`. 28Both speaker segmentation and embedding now run in pure PyTorch. This should ease deployment and possibly speed up inference. 29It requires pyannote.audio version 3.1 or higher.30 31It ingests mono audio sampled at 16kHz and outputs speaker diarization as an [`Annotation`](http://pyannote.github.io/pyannote-core/structure.html#annotation) instance:32 33- stereo or multi-channel audio files are automatically downmixed to mono by averaging the channels.34- audio files sampled at a different rate are resampled to 16kHz automatically upon loading.35 36## Requirements37 381. Install [`pyannote.audio`](https://github.com/pyannote/pyannote-audio) `3.1` with `pip install pyannote.audio`392. Accept [`pyannote/segmentation-3.0`](https://hf.co/pyannote/segmentation-3.0) user conditions403. Accept [`pyannote/speaker-diarization-3.1`](https://hf.co/pyannote-speaker-diarization-3.1) user conditions414. Create access token at [`hf.co/settings/tokens`](https://hf.co/settings/tokens).42 43## Usage44 45```python46# instantiate the pipeline47from pyannote.audio import Pipeline48pipeline = Pipeline.from_pretrained(49 "pyannote/speaker-diarization-3.1",50 use_auth_token="HUGGINGFACE_ACCESS_TOKEN_GOES_HERE")51 52# run the pipeline on an audio file53diarization = pipeline("audio.wav")54 55# dump the diarization output to disk using RTTM format56with open("audio.rttm", "w") as rttm:57 diarization.write_rttm(rttm)58```59 60### Processing on GPU61 62`pyannote.audio` pipelines run on CPU by default.63You can send them to GPU with the following lines:64 65```python66import torch67pipeline.to(torch.device("cuda"))68```69 70### Processing from memory71 72Pre-loading audio files in memory may result in faster processing:73 74```python75waveform, sample_rate = torchaudio.load("audio.wav")76diarization = pipeline({"waveform": waveform, "sample_rate": sample_rate})77```78 79### Monitoring progress80 81Hooks are available to monitor the progress of the pipeline:82 83```python84from pyannote.audio.pipelines.utils.hook import ProgressHook85with ProgressHook() as hook:86 diarization = pipeline("audio.wav", hook=hook)87```88 89### Controlling the number of speakers90 91In case the number of speakers is known in advance, one can use the `num_speakers` option:92 93```python94diarization = pipeline("audio.wav", num_speakers=2)95```96 97One can also provide lower and/or upper bounds on the number of speakers using `min_speakers` and `max_speakers` options:98 99```python100diarization = pipeline("audio.wav", min_speakers=2, max_speakers=5)101```102 103## Benchmark104 105This pipeline has been benchmarked on a large collection of datasets.106 107Processing is fully automatic:108 109- no manual voice activity detection (as is sometimes the case in the literature)110- no manual number of speakers (though it is possible to provide it to the pipeline)111- no fine-tuning of the internal models nor tuning of the pipeline hyper-parameters to each dataset112 113... with the least forgiving diarization error rate (DER) setup (named _"Full"_ in [this paper](https://doi.org/10.1016/j.csl.2021.101254)):114 115- no forgiveness collar116- evaluation of overlapped speech117 118| Benchmark | [DER%](. "Diarization error rate") | [FA%](. "False alarm rate") | [Miss%](. "Missed detection rate") | [Conf%](. "Speaker confusion rate") | Expected output | File-level evaluation |119| ------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- | --------------------------- | ---------------------------------- | ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |120| [AISHELL-4](http://www.openslr.org/111/) | 12.2 | 3.8 | 4.4 | 4.0 | [RTTM](https://huggingface.co/pyannote/speaker-diarization-3.1/blob/main/reproducible_research/AISHELL.SpeakerDiarization.Benchmark.test.rttm) | [eval](https://huggingface.co/pyannote/speaker-diarization-3.1/blob/main/reproducible_research/AISHELL.SpeakerDiarization.Benchmark.test.eval) |121| [AliMeeting (_channel 1_)](https://www.openslr.org/119/) | 24.4 | 4.4 | 10.0 | 10.0 | [RTTM](https://huggingface.co/pyannote/speaker-diarization-3.1/blob/main/reproducible_research/AliMeeting.SpeakerDiarization.Benchmark.test.rttm) | [eval](https://huggingface.co/pyannote/speaker-diarization-3.1/blob/main/reproducible_research/AliMeeting.SpeakerDiarization.Benchmark.test.eval) |122| [AMI (_headset mix,_](https://groups.inf.ed.ac.uk/ami/corpus/) [_only_words_)](https://github.com/BUTSpeechFIT/AMI-diarization-setup) | 18.8 | 3.6 | 9.5 | 5.7 | [RTTM](https://huggingface.co/pyannote/speaker-diarization-3.1/blob/main/reproducible_research/AMI.SpeakerDiarization.Benchmark.test.rttm) | [eval](https://huggingface.co/pyannote/speaker-diarization-3.1/blob/main/reproducible_research/AMI.SpeakerDiarization.Benchmark.test.eval) |123| [AMI (_array1, channel 1,_](https://groups.inf.ed.ac.uk/ami/corpus/) [_only_words)_](https://github.com/BUTSpeechFIT/AMI-diarization-setup) | 22.4 | 3.8 | 11.2 | 7.5 | [RTTM](https://huggingface.co/pyannote/speaker-diarization-3.1/blob/main/reproducible_research/AMI-SDM.SpeakerDiarization.Benchmark.test.rttm) | [eval](https://huggingface.co/pyannote/speaker-diarization-3.1/blob/main/reproducible_research/AMI-SDM.SpeakerDiarization.Benchmark.test.eval) |124| [AVA-AVD](https://arxiv.org/abs/2111.14448) | 50.0 | 10.8 | 15.7 | 23.4 | [RTTM](https://huggingface.co/pyannote/speaker-diarization-3.1/blob/main/reproducible_research/AVA-AVD.SpeakerDiarization.Benchmark.test.rttm) | [eval](https://huggingface.co/pyannote/speaker-diarization-3.1/blob/main/reproducible_research/AVA-AVD.SpeakerDiarization.Benchmark.test.eval) |125| [DIHARD 3 (_Full_)](https://arxiv.org/abs/2012.01477) | 21.7 | 6.2 | 8.1 | 7.3 | [RTTM](https://huggingface.co/pyannote/speaker-diarization-3.1/blob/main/reproducible_research/DIHARD.SpeakerDiarization.Benchmark.test.rttm) | [eval](https://huggingface.co/pyannote/speaker-diarization-3.1/blob/main/reproducible_research/DIHARD.SpeakerDiarization.Benchmark.test.eval) |126| [MSDWild](https://x-lance.github.io/MSDWILD/) | 25.3 | 5.8 | 8.0 | 11.5 | [RTTM](https://huggingface.co/pyannote/speaker-diarization-3.1/blob/main/reproducible_research/MSDWILD.SpeakerDiarization.Benchmark.test.rttm) | [eval](https://huggingface.co/pyannote/speaker-diarization-3.1/blob/main/reproducible_research/MSDWILD.SpeakerDiarization.Benchmark.test.eval) |127| [REPERE (_phase 2_)](https://islrn.org/resources/360-758-359-485-0/) | 7.8 | 1.8 | 2.6 | 3.5 | [RTTM](https://huggingface.co/pyannote/speaker-diarization-3.1/blob/main/reproducible_research/REPERE.SpeakerDiarization.Benchmark.test.rttm) | [eval](https://huggingface.co/pyannote/speaker-diarization-3.1/blob/main/reproducible_research/REPERE.SpeakerDiarization.Benchmark.test.eval) |128| [VoxConverse (_v0.3_)](https://github.com/joonson/voxconverse) | 11.3 | 4.1 | 3.4 | 3.8 | [RTTM](https://huggingface.co/pyannote/speaker-diarization-3.1/blob/main/reproducible_research/VoxConverse.SpeakerDiarization.Benchmark.test.rttm) | [eval](https://huggingface.co/pyannote/speaker-diarization-3.1/blob/main/reproducible_research/VoxConverse.SpeakerDiarization.Benchmark.test.eval) |129 130## Citations131 132```bibtex133@inproceedings{Plaquet23,134 author={Alexis Plaquet and Hervé Bredin},135 title={{Powerset multi-class cross entropy loss for neural speaker diarization}},136 year=2023,137 booktitle={Proc. INTERSPEECH 2023},138}139```140 141```bibtex142@inproceedings{Bredin23,143 author={Hervé Bredin},144 title={{pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe}},145 year=2023,146 booktitle={Proc. INTERSPEECH 2023},147}148```149