Modelpublic

BUT-FIT/DiCoP_v0.1

sourceHugging Facecc-by-4.0updated 2mo agoView on Hugging Face

3likes49downloads

Model Card

DiCoP — Diarization-Conditioned Parakeet

Target-speaker ASR built on nvidia/parakeet-tdt-0.6b-v2. Given audio and a diarization, it transcribes one speaker at a time.

The conditioning lives inside the encoder. Every frame is labelled silence / target / non-target / overlap (STNO), and each Conformer layer applies a small learned per-class transform — an FDDT block — before the layer runs. A whole meeting is decoded per speaker in one pass: no segmentation, no speaker embeddings, no separation front-end.


Base model	nvidia/parakeet-tdt-0.6b-v2
Parameters	618M
Encoder	24 × FastConformer, d_model 1024
Encoder frame rate	12.5 Hz (80 ms)
Vocabulary	1024 BPE tokens
Decoder	TDT (token-and-duration transducer)
Sample rate	16000 Hz

Usage

This checkpoint cannot be loaded by `nemo_toolkit` alone. Its encoder _target_ points at a class that lives in the DiCoP repository, and NeMo only resolves _target_s inside the nemo package unless that check is relaxed.

bash

git clone https://github.com/BUTSpeechFIT/DiCoP && cd DiCoP
pip install -r requirements.txt

python infer.py \
    --checkpoint BUT-FIT/DiCoP_v0.1 \
    --rttm /path/to/rttms/ --audio-dir /path/to/audio/ \
    --output hyp.stm

To drive the model directly:

python

import sys
sys.path.insert(0, "/path/to/DiCoP")

from utils.nemo import allow_external_nemo_targets, register_legacy_nemo_aliases

allow_external_nemo_targets()
register_legacy_nemo_aliases()

from src.model.asr_bpe_model import EncDecRNNTBPEModelSTNO

model = EncDecRNNTBPEModelSTNO.from_pretrained("BUT-FIT/DiCoP_v0.1")

transcribe() is deliberately disabled on this model. NeMo's transcription path cannot supply a mask, and an unconditioned encoder returns a fluent transcript of whoever is loudest — which looks correct but is not target-speaker output. Use infer.py, or transcribe_stno() with an STNO mask you build yourself (see src/data/stno.py).

Results

Oracle diarization, cpWER and tcpWER (collar 5s) in percent, whisper_nsf normalization applied. AMI's half-hour sessions used windowed local attention (-O model.encoder.self_attention_model=rel_pos_local_attn -O model.encoder.att_context_size=[256,256]) to bound memory; every other set is full-context, full-session.

Set	Sessions	cpWER	tcpWER
AMI-SDM dev / test	18 / 16	13.98 / 15.97	14.26 / 16.51
AMI-IHM-mix dev / test	18 / 16	11.20 / 11.75	11.41 / 12.15
NOTSOFAR-SDM dev1 / eval	177 / 160	17.44 / 17.56	17.93 / 17.94
LibriSpeechMix 2mix dev / test	2703 / 2620	2.62 / 2.54	2.62 / 2.54
LibriSpeechMix 3mix dev / test	2703 / 2620	6.79 / 6.34	6.80 / 6.35
Libri2Mix dev / test clean	3000	4.13 / 4.40	4.16 / 4.41
Libri3Mix dev / test clean	3000	30.93 / 33.12	31.00 / 33.19

Contact

If you have any questions, reach out to: iklement@fit.vut.cz