dys-asr/parakeet-rnnt-0.6b-all-syn-chunk-cutout

Parakeet RNN-T 0.6B — all SAP speech, synthetic, and chunked long audio

extraordinarylab/parakeet-unified-en-0.6b fine-tuned for English dysarthric speech recognition on every Speech Accessibility Project recording the constrained track allows, plus 103.1 hours of synthetic dysarthric speech and 78.8 hours recovered from recordings the duration cap would otherwise discard.

It is the sibling of `dys-asr/parakeet-rnnt-0.6b-sapc12-syn` and improves on it by 1.24 WER points and 0.91 CER points on the same held-out set. Four things changed at once, so the margin cannot be attributed to any one of them; the section on that is below.

No corpus outside the challenge is used, so this is a constrained-track model.

Usage

import soundfile as sf
import torch
from transformers import AutoProcessor, ParakeetForRNNT

model_id = "dys-asr/parakeet-rnnt-0.6b-all-syn-chunk-cutout"
device = "cuda" if torch.cuda.is_available() else "cpu"

processor = AutoProcessor.from_pretrained(model_id)
model = ParakeetForRNNT.from_pretrained(model_id).to(device).eval()

audio, rate = sf.read("utterance.wav", dtype="float32")
inputs = processor(audio, sampling_rate=rate, return_tensors="pt").to(device)
with torch.inference_mode():
    generated = model.generate(**inputs)
print(processor.batch_decode(generated, skip_special_tokens=True)[0])

Requires transformers>=5.9, which is where ParakeetForRNNT first appears. Audio must be 16 kHz mono.

Training data

The speaker total is below the column's sum because contributors recur across splits and because the synthetic voices are clones of real ones.

Duration and label-length filters leave 463,177 training examples: 0.5 to 45 seconds, and at most 200 label tokens. SAPC2 re-releases 182,575 of SAPC1's train recordings under the same filenames, and real SAP audio is deduplicated by filename so those are not silently given twice the weight. Square-bracketed spans, which in SAPC2 hold the interview prompt shown to the speaker rather than read aloud, are stripped before normalisation.

Chunked long audio

MAX_DURATION=45 discards 4,785 of these recordings — 88.2 hours, and 96% of them are Spontaneous Speech Prompts, so what the cap removes is disproportionately the connected speech that read prompts cannot stand in for. The audio is usable; it is the joint network's memory that sets the cap.

Each was force-aligned against its own transcript and cut at the pauses between words. 4,704 of the 4,785 aligned well enough to keep, recovering 78.8 hours in 8,933 chunks; the rest fell below an alignment-confidence floor and were left out. Three details decide whether this helps or hurts:

• —The aligner is a SAP-fine-tuned CTC checkpoint, not a stock English one. The speakers are dysarthric, and an aligner that has not heard that speech puts its boundaries in the wrong places on exactly the recordings that matter most.
• —A cut lands at the longest pause in the window between 40 s and the 45 s cap, rather than at the first boundary past 40 s — spending up to five seconds of chunk length to buy a quieter cut. Measured against each recording's own silence floor, cut points sit at 0.25 on a scale where 1.0 is that recording's speech level.
• —Padding of 0.40 s was settled by measurement, not taste. Decoding the chunks with a stock parakeet-tdt-0.6b-v3 and scoring them against their own transcripts gives a U-curve — 0.12 s: 9.08 WER, 0.25: 8.59, 0.40: 8.34, 0.60: 8.63, 0.90: 8.65. CTC emission is peaky, so an aligned word ends before the sound does and too little padding clips the tail off the chunk that owns it; too much reaches past the pause and pulls the neighbouring word's onset in, which no transcript on either side accounts for.

Chunk transcripts are carved out of the source text rather than the normalised text the alignment ran on. normalize_transcript is not idempotent — it folds a run of single letters into an acronym, so I, I my best friend normalises to I I MY BEST FRIEND and normalising that yields II MY BEST FRIEND. Storing normalised text would hand the training collator a transcript its own normaliser then corrupts, on 9.5% of the chunks here.

159 chunks cut from held-out dev recordings were removed from the training manifest. Without that step the evaluation set would have re-entered training through the long-audio back door.

Synthetic corpus

Generated with a fine-tuned Fun-CosyVoice3 in zero-shot voice-cloning mode and converted to 16 kHz mono PCM. Reference voices are real SAP recordings, sampled within etiology with probability proportional to the square root of the speaker's baseline CER, softly emphasising voices the stock recogniser finds difficult without concentrating the corpus on the handful of most severe speakers — who are also the ones most likely to make the TTS itself degenerate. Hours are allocated approximately as cerebral palsy 50%, Down syndrome 30%, stroke 17%, ALS 3%.

Synthetic speech can carry TTS artifacts, can imperfectly reproduce a speaker's impairment, and creates no independent speaker diversity when conditioned on voices already in the source corpus. It is not a clinical simulation and is not evidence about any diagnosis.

Training procedure

Ten epochs on sixteen GH200s, effective batch 32, AdamW at 1e-4 with a tri-stage schedule (10% warmup, 40% hold), weight decay 0.01, layerdrop 0.05, gradient clip 1.0, bf16, seed 42. Checkpoints ranked by CER on a 4,000-utterance subset of the held-out dev split; the released weights are epoch 9, the best of the ten.

Three augmentations run online in the dataloader, in this order:

SpecCutout is the one this family had not used before. SpecAugment's masks are bands: a time mask covers an interval across all 128 mel bins, a frequency mask covers a set of bins across all frames. A cutout rectangle covers a span of time and a span of frequency at once, so every frame and every bin stays represented somewhere else in the utterance and the hole has to be filled from two directions rather than one. Rectangles are drawn up to the configured extents rather than fixed at them, and are placed inside each utterance's real length rather than its padding — a rectangle landing past the end of a short utterance in a long batch would mask nothing, weakening the augmentation for exactly the utterances with the least signal to spare.

445 utterances were dropped for exceeding 200 label tokens. The cap is 200 rather than the 130 its predecessor used because a 40-second chunk of spontaneous speech runs to a median of 75 label tokens and a p99 of 199, where the read prompts run to a median of 12; at 130 a sixth of the chunked hours would have been discarded by the very filter the chunking exists to satisfy.

Evaluation

Greedy decoding, no language model, the project's transcript normaliser applied to references and hypotheses alike.

Per-epoch, on the 4,000-utterance subset used to rank checkpoints:

The curve is still descending at epoch 8 and flattens between 9 and 10, so the schedule is about the right length rather than comfortably long.

What the margin does and does not show

The predecessor is not a controlled comparison. Four things differ: this run adds SAPC1 dev (72.0 h), adds the chunked long audio (78.8 h), adds SpecCutout, and raises the label cap from 130 to 200. The 1.24 WER points are what all four bought together, and this release does not say how that splits. The label cap alone is not cosmetic — at 130 it would also have discarded long transcripts from the corpus the predecessor trained on.

The evaluation set

All 17,582 utterances (35.6 hours, ~141,000 reference words) from the 48 SAPC2 dev speakers who appear nowhere in training. Training includes SAPC1 dev, which shares 76 of SAPC2 dev's 124 speakers, so neither sapc1_dev nor the whole of sapc2_dev is a fair test here. Speaker overlap between the training manifest and this set is zero, verified by contributor ID.

The set is harder than SAPC2 dev as a whole: all 35 of SAPC2 dev's Parkinson's speakers also appear in SAPC1 dev, so it contains no Parkinson's speech, easily the lowest-error cohort. It breaks down as cerebral palsy 5,226 utterances, Down syndrome 4,750, ALS 4,606, stroke 3,000. Do not compare these numbers against results reported on the full dev split.

Numbers here come from the training loop's own final pass over the complete dev manifest, not from a subset.

Intended use and limitations

Research on atypical-speech recognition, and as a starting point for speaker-specific adaptation.

• —Requires `transformers>=5.9`. Not loadable by the 4.x line.
• —Slow relative to CTC. Decoding is autoregressive.
• —Lower-case, unpunctuated, numerals written as words. This differs from the CTC models in this family, which emit upper case. Lower case is not cosmetic: the tokeniser is overwhelmingly lower-case, so upper-case labels cost about 4.06 tokens per word against 1.67.
• —The four-way confound above. No ablation separates data from augmentation.
• —The held-out set excludes Parkinson's speech entirely, and is not representative of every etiology in training.
• —Single seed. No variance estimate, and no paired significance test behind the margin over the predecessor.
• —Chunk boundaries are machine-placed. 4 of the 4,704 chunked recordings lose one abbreviation-like token where per-token and whole-string normalisation disagree, and runs too short to stand alone and too far from a neighbour to merge are dropped with their audio rather than captioned onto a chunk that does not contain them.
• —English only, 16 kHz mono. The multilingual base was fine-tuned on English alone, and that capability is neither preserved nor evaluated.
• —Not a clinical tool. Nothing here supports inference about any diagnosis.

License and attribution

Released under CC BY 4.0, following the upstream NVIDIA Parakeet family. The Speech Accessibility Project corpora are governed by their own data use agreement and are not redistributed here; reproducing this training set requires authorized access. Fun-CosyVoice3's terms apply to the synthesis component.