dys-asr/parakeet-tdt-0.6b-syn

parakeet-tdt-0.6b-syn

nvidia/parakeet-tdt-0.6b-v3 fine-tuned for English atypical-speech recognition on real and synthetic dysarthric speech. The real training set joins the Speech Accessibility Project corpora, HeyJay! (ICPSR 39448), and one Google Project Relate takeout of UK ataxic speech. It is augmented with 103.1 hours of zero-shot CosyVoice speech conditioned on recordings from dysarthric speakers.

This is a token-and-duration transducer (TDT), not CTC. It has a prediction network, joint network, and duration head, and decoding is autoregressive.

Evaluation

On all 17,582 utterances from the 48 SAPC2 dev speakers who appear nowhere in the real training data:

Both scores use greedy decoding without a language model and the same transcript normaliser. Adding the synthetic corpus did not improve this benchmark: this checkpoint is 0.35 WER points and 0.27 CER points worse than the unconstrained checkpoint.

That is not a perfectly controlled synthetic-data ablation. This run accepts audio up to 45 seconds instead of 30 seconds and uses a different physical batch layout, although both runs have effective batch 32 and BatchNorm statistics over 16 examples. The result supports no claim that synthetic augmentation helps.

Read the evaluation set carefully

Training includes SAPC1 dev, which shares 28,253 recordings and 76 of SAPC2 dev's 124 speakers. Neither SAPC1 dev nor all of SAPC2 dev is therefore a fair test. The reported set contains only the 48 SAPC2 dev speakers absent from training (17,582 utterances, 126,993 reference words, 35.6 hours).

This subset has no Parkinson's speech: all 35 Parkinson's speakers in SAPC2 dev also appear in SAPC1 dev. It is consequently harder than SAPC2 dev as a whole, and its WER should not be compared directly with results reported on the full dev split.

Requirements and usage

`transformers>=5.9` is required. Earlier Transformers releases do not provide ParakeetForTDT.

import torch
from transformers import AutoProcessor, ParakeetForTDT

model_id = "dys-asr/parakeet-tdt-0.6b-syn"
processor = AutoProcessor.from_pretrained(model_id)
model = ParakeetForTDT.from_pretrained(model_id).eval()

# `audio` is a mono waveform sampled at 16 kHz.
inputs = processor(audio, sampling_rate=16_000, return_tensors="pt")
with torch.inference_mode():
    outputs = model.generate(**inputs)

text = processor.batch_decode(outputs.sequences, skip_special_tokens=True)
print(text)

Autoregressive TDT decoding is substantially slower than framewise CTC decoding—roughly an order of magnitude fewer samples per second in this model family.

Output text convention

The model emits lower-case, unpunctuated text with numbers written as words.

audio:  "lower the temperature three degrees"
output: lower the temperature three degrees

Lower-casing is part of training rather than display post-processing. The v3 tokenizer is overwhelmingly lower-case; capitalized labels fragment into many more tokens. Evaluation normalises both references and hypotheses consistently.

Training data

The input manifest contains 469,430 recordings (1,023.2 hours). Duration and label-length filters leave 463,740 training examples.

SAPC2 republishes much of SAPC1. Real SAP recordings are deduplicated by audio filename so those utterances are not silently upweighted. Square-bracketed interview prompts, which were shown but not spoken, are removed from labels.

The underlying corpora have separate access terms. The Speech Accessibility Project data is governed by its data-use agreement; HeyJay! and the Project Relate takeout carry their own terms. Publishing model weights does not publish the source recordings, but reproducing this training set requires authorized access to each corpus.

Synthetic corpus

The synthetic set was generated with a fine-tuned Fun-CosyVoice3 model in zero-shot voice-cloning mode and converted to 16 kHz mono PCM. Its reference voices come from real Speech Accessibility Project recordings. Reference speakers are sampled within etiology with probability proportional to the square root of their baseline CER, softly emphasizing voices the stock ASR model finds difficult.

The text mix is 30% uncommon existing SAP transcripts, 50% novel SAP-like sentences, and 20% sentences targeted at errors made by the baseline recognizer. The manifest retains whether each example is existing, novel, or error-targeted, along with its reference speaker and that speaker's baseline CER.

Synthetic speech can contain TTS artifacts, can imperfectly reproduce a speaker's impairment, and does not create independent speaker diversity when it is conditioned on voices already represented in the source corpus. It must not be treated as a clinical simulation or as evidence about any diagnosis.

Training hyperparameters

Model weights remain float32 under bf16 autocast. Loading the checkpoint itself in bfloat16 can fail because the feature extractor emits float32 and the first convolution requires matching dtypes.

WER on the fixed 4,000-utterance development subset by epoch:

Epoch 10 is the released checkpoint. On the full held-out split it scores 10.87% WER and 6.72% CER.

Limitations

• —Requires transformers>=5.9 and autoregressive decoding is slow relative to CTC.
• —Produces lower-case, unpunctuated English with numerals written as words.
• —The multilingual base model was fine-tuned only on English, so its original multilingual capability is not preserved or evaluated.
• —The synthetic corpus did not improve the held-out benchmark relative to the closest real-data model.
• —The held-out set excludes Parkinson's speech and is not representative of all etiologies in training.
• —HeyJay!, Project Relate, and the synthetic reference voices all appear in training, leaving no clean out-of-corpus evaluation for those sources.
• —Synthetic samples may contain voice-cloning artifacts and should not be used as clinical representations of disability.
• —Results are from one training seed, with no variance estimate or per-speaker breakdown.