dys-asr/parakeet-tdt-0.6b-syn-soup

Parakeet TDT 0.6B — Synthetic Model Soup

dys-asr/parakeet-tdt-0.6b-syn-soup is an English automatic speech recognition model adapted for dysarthric and ataxic speech. It is a precomputed, equal-weight model soup of three Parakeet TDT 0.6B fine-tunes:

• —`dys-asr/parakeet-tdt-0.6b-all-aug`
• —`dys-asr/parakeet-tdt-0.6b-unconstrained`
• —`dys-asr/parakeet-tdt-0.6b-syn`

All three ingredients descend from `nvidia/parakeet-tdt-0.6b-v3`. Their floating-point parameters were averaged uniformly, with weight 1/3 per checkpoint. Non-floating tensors were combined with an elementwise maximum. The result is stored here as one ordinary checkpoint: model averaging is not performed during inference and adds no inference-time memory or compute cost.

Because the unconstrained ingredient uses external dysarthric-speech corpora, this model should be treated as an unconstrained-track model.

Evaluation

The primary metric is character error rate (CER). The following decoding comparison used a 1,500-utterance subset of held-out SAPC2 development speakers, both available references, and the official challenge text normalizer. No external language model was used.

Beam size 2 is the recommended setting when optimizing primarily for CER. Beam search is a separate decoding implementation and is not embedded in these weights. In particular, ParakeetForTDT.generate(num_beams=...) does not provide this custom TDT beam search. The associated Codabench submission uses the implementation in codabench/parakeet_tdt_v3_syn_soup_beam/tdt_beam.py from the ataxia repository and defaults to SAPC_BEAM_SIZE=2.

For provenance, the individual ingredients produced the following results on the full held-out SAPC2 development set (17,582 utterances from 48 speakers):

These full-set ingredient scores are not directly comparable with the soup decoding table above because the latter uses a different 1,500-utterance evaluation subset.

Usage

The checkpoint uses the Hugging Face Transformers implementation of Parakeet TDT and requires a Transformers release that includes ParakeetForTDT (transformers>=5.9). Audio should be mono, sampled at 16 kHz.

import torch
from transformers import AutoProcessor, ParakeetForTDT

model_id = "dys-asr/parakeet-tdt-0.6b-syn-soup"
device = "cuda" if torch.cuda.is_available() else "cpu"

processor = AutoProcessor.from_pretrained(model_id)
model = ParakeetForTDT.from_pretrained(model_id).to(device).eval()

# `audio` is a one-dimensional 16 kHz waveform.
inputs = processor(audio, sampling_rate=16_000, return_tensors="pt")
inputs = {name: value.to(device) for name, value in inputs.items()}

with torch.inference_mode():
    outputs = model.generate(**inputs)

transcript = processor.batch_decode(
    outputs.sequences,
    skip_special_tokens=True,
)[0]
print(transcript)

This example performs greedy decoding. Use the Codabench TDT beam-search implementation referenced above for the reported beam-search results.

The training transcripts were normalized to lower-case, unpunctuated text with numbers written as words. Applications should expect output in that style.

Training data represented by the soup

The three ingredient models collectively represent:

• —SAPC1 train and development data plus deduplicated SAPC2 training data.
• —Online speed perturbation and SpecAugment in the all-augmentation recipe.
• —HeyJay! (ICPSR 39448) and Project Relate UK ataxic-speech data in the unconstrained recipe.
• —55,988 synthetic CosyVoice utterances (approximately 103.1 hours) in the synthetic recipe. The synthetic voices were conditioned on SAP recordings; this is voice conditioning, not a validated clinical simulation of dysarthria.

Refer to the three ingredient model cards for detailed filtering, duration limits, corpus counts, and source-specific access conditions.

Model soup construction

For each floating-point tensor with matching name and shape, the checkpoint stores

(all_aug + unconstrained + syn) / 3

For non-floating tensors such as counters, it stores the elementwise maximum. Configuration, tokenizer, processor, and generation metadata were copied from the first compatible ingredient. The ingredients share the same architecture and base model.

Intended use

This model is intended for research and evaluation of English ASR for atypical, dysarthric, and ataxic speech, including unconstrained-track challenge submissions. It is not a diagnostic system, clinical decision aid, or substitute for professional assessment.

Limitations

• —The beam-search measurements use a 1,500-utterance subset and may not generalize to the complete evaluation set or other corpora.
• —External-data and synthetic-data gains may vary by speaker, condition, recording setup, and severity.
• —Synthetic speech may introduce artifacts or underrepresent clinically important variation.
• —The source corpora and evaluation data do not represent every diagnosis, accent, language, age group, or communication style.
• —Fine-tuning for normalized English transcription may reduce the multilingual, punctuation, capitalization, or timestamping capabilities of the base model.
• —Recognition errors can disproportionately affect people whose speech differs from the training distribution. Human review is recommended for consequential uses.

License and data terms

The NVIDIA base model is released under CC BY 4.0. The fine-tuning ingredients incorporate datasets governed by separate access and data-use terms, so this repository is marked license: other. Users are responsible for reviewing and complying with the base-model license and all applicable source-corpus terms before redistribution or deployment.

Citation

When using this checkpoint, please cite the original Parakeet model and the relevant SAP Challenge and source-corpus publications. See the base and ingredient model cards linked above for their citation details.