dys-asr/parakeet-tdt-0.6b-unconstrained-soup

parakeet-tdt-0.6b-unconstrained-soup

A uniform weight average -- a model soup -- of two nvidia/parakeet-tdt-0.6b-v3 fine-tunes:

• —`dys-asr/parakeet-tdt-0.6b-all-aug`, trained on every Speech Accessibility Project recording;
• —`dys-asr/parakeet-tdt-0.6b-unconstrained`, the same plus HeyJay! (ICPSR 39448) and a Project Relate takeout of UK ataxic speech.

Same base model, same recipe, same seed, ten epochs each. Averaging costs nothing at inference: the soup is one model of the same size, not an ensemble.

Because one ingredient trained on corpora outside the challenge, this model belongs in the Speech Accessibility Project Challenge's unconstrained track, as its name says.

It was expected to lose

All three rows below are the same 17,582 utterances from 48 speakers, no language model, both sides through the same normaliser, all scored by the same evaluator:

The prior said otherwise. Every CTC soup in this family is beaten by the best single CTC model on this exact set -- 11.65% against 11.89%, 12.12% and 12.30% -- and those soups only won on corpora the models had never seen. The metric here is in-domain, so the expectation was a loss and the measurement was a win: 308 fewer word errors than either ingredient.

The difference between the two cases is probably how far apart the ingredients are. The CTC soups mixed models trained on different corpora; these two differ by 2.4% of their training records and agree almost everywhere.

Do not compare these numbers with the ones on the ingredient model cards. Those cards report 10.73% and 10.52%, from the training loop's own dev pass, which applies the training duration filter and therefore scores 17,319 utterances -- dropping the 263 recordings over 30 s, which are among the hardest. The rows above score all 17,582. The offset is consistent, so either set of numbers ranks the models correctly; mixing them does not.

There is no paired significance test behind the 0.24-point margin. The evaluations did not retain per-utterance predictions. On 126,993 reference words the unpaired standard error of one such WER is about 0.09 points, which puts the margin near 2.7 of them; a paired interval would be tighter, but it was not computed.

Requirements

`transformers>=5.9`. ParakeetForTDT does not exist before it.

import torch
from transformers import AutoProcessor, ParakeetForTDT

model_id = "dys-asr/parakeet-tdt-0.6b-unconstrained-soup"
processor = AutoProcessor.from_pretrained(model_id)
model = ParakeetForTDT.from_pretrained(model_id).eval()

inputs = processor(audio, sampling_rate=16_000, return_tensors="pt")  # 16 kHz mono
with torch.inference_mode():
    outputs = model.generate(**inputs)
print(processor.batch_decode(outputs.sequences, skip_special_tokens=True))

Decoding is autoregressive rather than an argmax over frames, so it is substantially slower than the CTC siblings.

Output text convention

Numbers as words, lower-case, unpunctuated, inherited from both ingredients. Lower case is not cosmetic: v3's tokeniser is overwhelmingly lower-case, so upper-case labels cost 4.06 tokens per word against 1.67. Note this differs from the CTC siblings, which emit upper case.

How it was made

Uniform average, no greedy selection -- with two ingredients there is nothing to select. Float tensors are averaged in float64 and cast back; BatchNorm's int64 num_batches_tracked counter is carried across as a maximum rather than averaged, since averaging a counter is meaningless and would silently floor.

Everything runs on the CPU on purpose. cuDNN flattens the prediction network's LSTM weights into one buffer the first time it runs on CUDA, leaving decoder.lstm.weight_ih_l0 and its siblings sharing storage; safetensors writes that without complaint and then refuses to read it back. The build script reloads what it wrote and compares every tensor before treating the soup as valid.

Training data

The union of the ingredients', which is the second one's: SAPC1 train and dev, SAPC2 train, HeyJay! (ICPSR 39448) and a Project Relate takeout -- 413,442 recordings, 920.1 hours, 1,094 speakers, filtered to 0.5-30 s and labels of at most 130 tokens. The ingredient cards carry the per-corpus breakdown.

The Speech Accessibility Project data is governed by its own data use agreement, and HeyJay! and the Project Relate takeout each carry separate terms.

Read the evaluation set carefully

Training includes SAPC1 dev, which shares 76 of SAPC2 dev's 124 speakers, so neither sapc1_dev nor the whole of sapc2_dev is a fair test. The set above is the 48 SAPC2 dev speakers appearing nowhere in training. It is harder than SAPC2 dev as a whole: all 35 of SAPC2 dev's Parkinson's speakers also appear in SAPC1 dev, so it contains no Parkinson's speech, easily the lowest-error cohort.

Limitations

• —Requires `transformers>=5.9`. Not loadable by the 4.x line.
• —Slow relative to CTC. Autoregressive decoding.
• —Lower-case, unpunctuated, numerals as words, unlike the CTC siblings.
• —No paired significance test behind the margin over its ingredients.
• —A soup of two, the smallest there is. A third ingredient -- a second seed, or layerdrop 0.0 -- is untested.
• —No out-of-corpus evaluation is possible: HeyJay! and the Project Relate takeout are entirely inside one ingredient's training data.
• —The held-out set excludes Parkinson's speech entirely.
• —Single seed per ingredient. No variance estimate.