kvizas/parakeet-tdt-0.6b-diar-adapters

Parakeet diarization adapters — a controlled four-way comparison

License. Everything in this repository — the four adapter checkpoints, the four head checkpoints, diarize.py and the configuration files — is released under CC-BY-4.0, the same licence as the base model `nvidia/parakeet-tdt-0.6b-v2`, which this repository does not redistribute: diarize.py downloads it from NVIDIA at run time.

Base model version. These were trained on Parakeet-TDT-0.6B v2 (nvidia/parakeet-tdt-0.6b-v2), not v3. v3 shares the architecture, so diarize.py runs unchanged against it, but these weights were fitted to v2's layer-5 representations and are not expected to transfer.

Integration pitfall (found in a production benchmark, 2026-09-21). NeMo's model.transcribe() returns with the encoder in training mode (it calls encoder.unfreeze(partial=True), which calls module.train()), so if you run ASR with transcribe() and then read layer-5 features for this head from the same model object, the head sees dropout-corrupted, non-reproducible features. Call model.eval() (or use diarize.py, which never calls transcribe()) before extracting features.

2026-09-25 update — the project's best system: bottleneck64_full/

The four folders below are the original controlled comparison (short budget, tapped-feature linker). A fifth folder, `bottleneck64_full/`, is the same bottleneck-64 recipe trained at the full stage-2 budget (30 epochs × 80 h of AMI train, ≈ one pass per epoch) and run with TitaNet-large voice-embedding speaker profiles for cross-window linking (--embed-model titanet_large, new in this diarize.py) at the head's training window length (180 s / 90 s hop). It is the best diarization system of the project and supersedes the "not the best system" note further down.

AMI dev, IHM-Mix, 18 meetings, same scorer; every row below was scored on one machine (the original Phase-2 release re-scores at 36.8 % there, vs 32.9 % when first measured — the encoder is not bit-reproducible between processes and one flipped link costs a meeting):

cpWER is the word error rate after speaker attribution, all 18 meetings, with the same Parakeet transcript (chunked 60 s / 5 s long-form) for every row — only the diarization differs. The shared-encoder system beats the cascade on it for the first time, linking with a 25 M-parameter speaker model instead of a 117 M-parameter diarizer, and WER is unchanged.

python diarize.py --audio meeting.wav --out meeting.rttm \
    --head bottleneck64_full/head.pt --adapters bottleneck64_full/adapters.pt \
    --postproc bottleneck64_full/postproc.json \
    --window-sec 180 --hop-sec 90 --embed-model titanet_large

--embed-model titanet_large makes NeMo download TitaNet-large (~100 MB) once. Without it the same checkpoints use the tapped-feature profiles and score 29.4 % (180/90) / 28.0 % (300/150). The flag works with every folder here and with the Phase-2 release: on the Phase-2 head alone it is worth 12 DER points, with no training. Tested negatives from the same night, so nobody repeats them: a powerset (speaker-set softmax) objective, down-weighting the scorer's collar in the loss, joint clustering of all windows, and a second linking pass — none beat this configuration. Full numbers: bottleneck64_full/config.json.

2026-09-26, evening — rob_bn64/: the recommended system

Use `rob_bn64/`. It is pool_bn64 continued for 20 epochs with on-the-fly acoustic augmentation: speech from another recording mixed in 8-20 dB below the target without labels (so background chatter is taught to be non-speech), coloured noise, synthetic reverberation, a telephone band-pass and random gain. It is run with the voice re-segmentation below switched on and allowed up to 8 voices. Session DER, same scorer for every cell (md-eval, collar 0.25 s, overlap scored), post-processing chosen on validation only:

(The two older columns are those systems as released, without re-segmentation.) The café row is the reason for the augmentation: false-alarm time on background chatter fell from 5 858 s to 428 s. Speech recognition is untouched by construction.

python diarize.py --audio rec.wav --out rec.rttm \
    --head rob_bn64/head.pt --adapters rob_bn64/adapters.pt --postproc rob_bn64/postproc.json \
    --window-sec 180 --hop-sec 90 --embed-model titanet_large

--reseg on and --reseg-max-speakers 8 are now the defaults. An earlier version of the re-segmentation capped the voice count before discarding outlier clusters, so a few noisy chunks could survive as "speakers" while real voices were merged; that bug is what made it look harmful on long meetings. With the fix it is worth 5 to 37 points on every long-recording set (ICSI 49.4 → 12.6 %). Also tried and not adopted: an eight-slot head (rob8_bn64; better on AMI test by one point, worse on validation and on the short clips). Every number and ablation: rob_bn64/config.json.

2026-09-26 — pool_bn64/: the same recipe trained on 400 h of real data

bottleneck64_full was trained on AMI alone (80 h of headset-mixed meetings) and it showed: strong on AMI, weak everywhere else. `pool_bn64/` continues that exact system (warm start from its head and adapters) for 30 epochs × 100 h drawn from a 400 h pool — AMI headset mix and distant mic, ICSI, NOTSOFAR-1, VoxConverse dev, MSDWild, LibriCSS, plus simulated mixtures at a low weight — with short clips packed into 180 s windows and windows of more than four active speakers skipped. The epoch was chosen on AMI dev and MSDWild val scored together, so the selection is not driven by one corpus. Session DER, TitaNet linker, 180 s / 90 s, same scorer for every cell, no re-segmentation:

One point on AMI dev, within run-to-run noise, 2 points better on AMI test, and 10 to 24 points better everywhere else. Transcripts are unchanged (LibriSpeech dev-clean WER 1.98 % / 1.84 % with these adapters vs 1.98 % / 1.95 % without, full / local attention); on speaker-attributed AMI transcripts it is 2 cpWER points behind bottleneck64_full (45.0 vs 43.0, cascade 44.2). Full provenance and every number in pool_bn64/config.json.

Which folder to use. pool_bn64/ for meetings and in-the-wild recordings of any length and language; bottleneck64_full/ with --reseg on for short, clean, few-speaker clips. On a company benchmark of such clips (17 synthetic conversations, 23 s to 20 min) the AMI-only head plus forced re-segmentation scores 6.1 % DER against 16.7 % for the pooled head, because the pooled head misses more speech on some single-speaker clips. Neither handles background chatter: on the same benchmark's cafe condition both exceed 100 % DER from false alarms (the pooled one worse, 2.4× the false-alarm time, a side effect of in-the-wild training data where background voices are labelled). That is the open problem, and it is a speech/non-speech one, not an identity one.

A re-sweep of the post-processing on the pooled model's own validation features chose a shorter median filter that was better window-locally and worse at session level on every set, so the inherited 0.4 / 21 / 0.5 is shipped (negative result, config.json).

python diarize.py --audio rec.wav --out rec.rttm \
    --head pool_bn64/head.pt --adapters pool_bn64/adapters.pt --postproc pool_bn64/postproc.json \
    --window-sec 180 --hop-sec 90 --embed-model titanet_large

(Superseded: re-segmentation is now `--reseg on` by default, see the `rob_bn64` section.) Previously --reseg auto: it ran only on recordings that fit in one window. On long meetings it undoes the linker's work with this head (AMI dev 21.8 → 33.2 %, VoxConverse 25.9 → 29.7 %); on short clips it is what makes the difference.

2026-09-25, later — voice re-segmentation (--reseg) for short recordings

On a company benchmark of short synthetic conversations (ElevenLabs voices, 23 s to 20 min, 2–4 speakers, no overlap) the system above was the worst provider. The reason turned out to be specific: a clip shorter than one window never reaches the linker, so the head alone decides who is who — and the head, trained on AMI headset mixes only, folds a third or fourth clean voice into the first two slots (their posteriors never rise above 0.3). Two-speaker clips were near perfect; every 3- or 4-speaker clip collapsed to two speakers.

--reseg (default auto, see above; on forces it, off disables it) fixes that without training: after linking, every segment is cut into ~2 s chunks, each chunk is embedded with TitaNet, the chunks are clustered by voice (average linkage on cosine distance, threshold 0.7, at most 4 speakers, clusters with under 5 s of speech folded into the nearest one), and the clusters become the speakers. Chunks that overlap another speaker's segment are not embedded and keep their segment's label. The head still decides when there is speech; TitaNet decides who. It is the pyannote recipe applied on top of this head.

python diarize.py --audio clip.wav --out clip.rttm \
    --head bottleneck64_full/head.pt --adapters bottleneck64_full/adapters.pt \
    --postproc bottleneck64_full/postproc.json \
    --window-sec 180 --hop-sec 90 --embed-model titanet_large --reseg on

That benchmark's clean condition, 17 clips (6 of them single-speaker), DER with the project scorer: 31.0 % → 6.1 %; its 11 multi-speaker clips 33.2 % → 7.1 %. Far-field 36.9 → 31.8, telephone 31.3 → 24.8. The cafe and "worst case" conditions stay above 100 % DER with or without the flag: there the head fires on the background chatter (false alarm), which is a speech/non-speech robustness problem, not an identity one. Whether --reseg helps on long meetings (AMI dev) is reported in bottleneck64_full/config.json under reseg.

What an adapter is

The companion release `kvizas/parakeet-tdt-0.6b-diar-head` does speaker diarization by reading one activation out of a completely frozen ASR encoder — Conformer layer 5, one 1024-dimensional vector every 80 ms — and feeding it to a small trained transformer head. The encoder is never trained; it is a fixed feature extractor, and the head has to make do with whatever it happens to provide.

An adapter is the obvious next question: could the encoder be nudged towards diarization without retraining it? A full fine-tune of 600 M parameters is out of reach on a free Colab GPU, would need far more data than AMI's training set, and would destroy the property that made the system interesting — that transcription quality is untouched. An adapter buys a middle option. It is a small trainable module placed beside or after a frozen block, initialised so that it does exactly nothing on the first step, and then trained while every pretrained weight stays exactly where it was. Two kinds are in this repository:

• —Bottleneck adapter (Houlsby et al., 2019) — a two-layer network whose output is added to a frozen layer's output: y = y + up(gelu(down(y))), with down squeezing 1024 dimensions to 64 (or 256) and up expanding them back. up starts at exactly zero, so at step 0 the encoder is bit-for-bit the original one.
• —LoRA (Hu et al., 2021) — a thin trainable detour beside one frozen weight matrix: y = W x + (α/r)·B(A x), with rank r = 8 out of 1024, placed on the attention query and value projections. B starts at zero, so again the model begins unchanged. The bet is that the useful adaptation of a big pretrained matrix is low-rank.

Both live in the bottom five encoder layers only, because the head taps layer 5 and anything above it receives no gradient. Neither modifies a single pretrained number: the checkpoints here contain only the adapters, and attaching them is done with a forward hook or a wrapper module at load time.

What this release is

A controlled comparison, not four independent models. All four arms were trained with an identical recipe — same initial head, same data, same number of epochs, same window sequence, same learning rates, same seed — and differ only in what was attached to the encoder. That is what makes the rows below comparable with each other. The control arm has zero adapter parameters: it is the same fine-tune with nothing attached, and it answers "how much of any gain is simply the extra training?"

Each folder holds head.pt (the head co-trained with those adapters), adapters.pt, postproc.json and a config.json with that arm's numbers and provenance. A head and its adapters belong together — pass the pair, or neither.

Results

AMI dev, IHM-Mix, all 18 meetings (9.67 h), meeteval 0.4.3 / md_eval_22, collar 0.25 s, profile linker at weight 1.0, post-processing 0.4 / 21 frames / 0.5 s frozen across all four arms. WER is LibriSpeech dev-clean, first 200 utterances (3 639 reference words), measured twice: with NeMo's default full attention and with the limited-context attention the diarizer actually runs under.

"300/150" and "180/90" are the analysis window and hop in seconds; both are legitimate settings of the same system and are reported because they disagree.

Use `bottleneck64`. It is the only configuration that beats the control at both window settings — by 1.7 and 3.7 points — and the gain is almost entirely in the speaker-error term (4 728 s → 3 873 s at 180/90), which is where the diarization problem actually lives. 660 800 parameters, 0.11 % of the frozen encoder, buy that.

lora8 and bottleneck256 each win at one window setting and lose at the other: LoRA-8 is 3.5 points better at 180/90 and 1.7 worse at 300/150; bottleneck-256 is 4.1 better at 300/150 and 0.9 worse at 180/90. That pattern is inside the ±2-point noise of this benchmark — re-extracting the encoder features alone, with identical weights, moves the 18-meeting aggregate by about two points — so neither has shown anything. They are published because a controlled comparison that hides its inconclusive arms is not one.

On word error rate: no measurable cost. Every arm lands between 1.90 % and 1.98 %, the same band as the untouched encoder. The whole spread is two or three words out of 3 639 — at this test size it bounds the effect at "smaller than we can see", nothing stronger. Even the 2.6 M-parameter bottleneck256 is inside it.

This four-way table is not the project's best diarization system (see the 2026-09-25 update above)

Until 2026-09-25 the best system was the Phase-2 release, `kvizas/parakeet-tdt-0.6b-diar-head`, at 33 ± 2 % session DER. The comparison here starts from the earlier, stage-1 head on purpose — every attempt to push the already-AMI-tuned head further, with or without adapters, made it worse — so all four arms sit at a lower starting point than that release. Read this repository as a clean measurement of what adapters do, not as a leaderboard entry. If you want the best diarization, take the Phase-2 model; if you want to see whether adapters help, take these. As of 2026-09-25 the answer to both is bottleneck64_full/ with --embed-model titanet_large.

Usage

pip install -r requirements.txt
python diarize.py --audio meeting.wav --out meeting.rttm \
    --head bottleneck64/head.pt \
    --adapters bottleneck64/adapters.pt \
    --postproc bottleneck64/postproc.json

Swap the folder name for any of the four. The audio must be 16 kHz mono (wav or flac); multi-channel files are averaged, other sample rates are resampled only if torchaudio or librosa is installed. --device is auto|cpu|mps|cuda, and --window-sec 180 --hop-sec 90 gives the second column of the table. The script prints one line saying what it attached:

[diarize] attached bottleneck adapters (bottleneck 64) to encoder layers 1-5: 660,800 parameters from bottleneck64/adapters.pt

control/adapters.pt is a valid file that attaches nothing, and says so. Passing no --adapters at all runs the plain frozen encoder, which is exactly the Phase-2 pipeline.

From Python:

from diarize import diarize

for start, end, speaker in diarize("meeting.wav",
                                   head="bottleneck64/head.pt",
                                   adapters="bottleneck64/adapters.pt",
                                   postproc="bottleneck64/postproc.json"):
    print(f"{start:8.2f} {end:8.2f} {speaker}")

The output is an RTTM in the NIST SPEAKER format with labels spk0…spk3:

SPEAKER meeting 1 34.240 2.880 <NA> <NA> spk0 <NA> <NA>

diarize.py here is the same file as in the Phase-2 release with adapter support added; with no adapters it behaves identically, verified segment for segment on the same audio.

How they were trained

One recipe, four arms, one seed each. Initialise the head from the Phase-2 stage-1 checkpoint (trained on simulated LibriSpeech conversations plus AMI); train on AMI train IHM-Mix only, no simulated speech; 24 epochs × 40 sampled hours (960 audio-hours, 4 800 optimizer steps); batches of 4 × 180 s windows drawn at random session and random offset; one AdamW over both parameter groups with adapter lr 1e-4 and head lr 5e-5, weight decay 0.01, 10 % linear warmup then cosine to zero, grad-clip 1.0; bf16 autocast. The checkpoint kept is the best per-epoch window-local DER over all 18 dev meetings. Each run takes about an hour on a free-tier Colab L4.

The head reads the adapted layer-5 output: the adapters are attached before the tap hook, so gradients flow head → layer 5 → adapters, and the forward pass is truncated at layer 5 during training because layers 6–24 are never read.

A negative result worth knowing

Before this comparison, four earlier runs warm-started the adapters from the AMI-fine-tuned head instead of the stage-1 one, and every one of them got worse from the first epoch onward — even with the head frozen, and even on AMI-only data. The diagnosis was not a broken training loop (an lr = 0 control was bit-stable) but a genuine disagreement between objective and metric: held-out frame BCE fell from 0.3742 to 0.3582 and frame F1 rose at every threshold, while collar-scored DER rose at every threshold — the adapters moved the activations to slightly wrong moments, which frame-level BCE tolerates and DER charges for. The full diagnostics are in the project log; the practical lesson is that once you let a model change the features rather than only the read-out, a frame-level proxy objective stops being safe.

Limitations

• —Dev set only. Every number here is AMI dev. No held-out test result exists, and the checkpoints, thresholds and window lengths were all selected on those same 18 meetings.
• —One seed per arm. Against ±2-point aggregate noise, the bottleneck64 vs bottleneck256 vs lora8 ordering is not resolved. Three seeds would settle it.
• —±2 points of noise. Re-extracting the encoder features with identical weights moves the 18-meeting aggregate by about two points; per-meeting DER swings by ±5, because one flipped linking decision propagates to the end of a meeting.
• —Four speakers, hard limit. Four output slots and a 4×4 assignment: a fifth person gets merged away.
• —English meeting speech. AMI's mixed-headset channel and LibriSpeech-derived simulated conversations. Nothing here has seen telephone speech, another language, or a far-field array.
• —The encoder is not bit-reproducible. Two feature extractions of the same audio give slightly different activations, on the same machine as well as across GPUs.
• —Adapters only below the tap. Layers 1–5, because anything above layer 5 receives no gradient from a head that reads layer 5. This says nothing about upper-layer adaptation.
• —The WER bound is weak. 200 short, clean, read-speech utterances bound the easy case only; they say nothing about what adapters do to a 50-minute meeting under limited-context attention, which is the regime the diarizer runs in.
• —Not a streaming system, and the linker never revisits a decision.

Code and paper

This is the Phase-3 result of a coursework research project on extending a frozen ASR encoder to diarization. The code lives in a private repository (Kvizas/kursinis); available on request. diarize.py is a self-contained copy of the inference path, including the adapter implementation.

If you use this, please also cite the base model, `nvidia/parakeet-tdt-0.6b-v2`.