dys-asr/parakeet-ctc-0.6b-soup

parakeet-ctc-0.6b-soup

A model soup: the weights of two fine-tunes averaged into one network, at 1 part `dys-asr/parakeet-ctc-0.6b-sapc2` to 2 parts `dys-asr/parakeet-ctc-0.6b-all`. No training was done here. Inference costs exactly what one model costs.

All rows below are the same 17,582 utterances from 48 speakers, greedy CTC decoding, no language model, both sides through the same normaliser:

What the soup does and does not buy

It is worth 0.19 WER points over the best ingredient. That is a real but small gain, and two things should temper it:

• —The mixing weight was chosen on the set reported here. Three combinations were tried -- all three models at 1:1:1, then sapc2+all at 1:1 and at 1:2 -- and the best was kept. Choosing and reporting on one set flatters the number, so treat 11.89% as the optimistic end. The CER moving with it (7.02% to 6.89%) is the more convincing part, since CER is estimated from far more units.
• —Not every soup helps. Averaging all three models made things worse than the best ingredient -- 12.30% against 12.08% -- because sapc1 at 14.65% drags the average down. Soups reward ingredients of similar quality; these differ by training data rather than by seed or hyperparameter, which is not the setting soups were designed for.

Read the evaluation set carefully

parakeet-ctc-0.6b-all trains on SAPC1 dev, which shares 28,253 recordings and 76 of SAPC2 dev's 124 speakers, so neither sapc1_dev nor the whole of sapc2_dev is a fair test for this soup. What is fair is the 48 SAPC2 dev speakers appearing nowhere in any ingredient's training data, and that is the set used above.

That set is harder than SAPC2 dev as a whole: all 35 of SAPC2 dev's Parkinson's speakers also appear in SAPC1 dev, so it contains no Parkinson's speech, and Parkinson's is the easiest cohort here by a wide margin. Do not compare this model's 11.89% against parakeet-ctc-0.6b-sapc2's published 11.51% on all of SAPC2 dev. They are different sets. Scored head to head, as above, the soup wins by 1.46 points.

Reproducing it

python -m ataxia.cli soup \
  --models dys-asr/parakeet-ctc-0.6b-sapc2 dys-asr/parakeet-ctc-0.6b-all \
  --weights 1 2 \
  --output-dir parakeet-ctc-0.6b-soup

Float tensors, including the BatchNorm running_mean and running_var buffers, are averaged in float64 and cast back. The 24 int64 num_batches_tracked counters are not averaged -- a counter has no meaningful mean, and integer division would floor it -- and are carried across as a maximum. This changes nothing at inference: that counter is read only when a BatchNorm layer sets momentum=None, and Parakeet's layers set a momentum.

Output text convention

This model writes numbers as words and emits upper-case, unpunctuated text, inherited from its ingredients.

audio:  "lower the temperature three degrees"
output: LOWER THE TEMPERATURE THREE DEGREES     # not "... 3 DEGREES"

The tokeniser has 1,025 tokens and no digit characters. If you score this model, verbalise your references the same way or numerals will dominate your errors.

Results by speaker

Pooled over words the figure is 11.89%; averaged with equal weight per speaker it is 14.73%, and the spread is wide:

20 of the 48 speakers are under 10% WER; 3 are above 30%.

Usage

import torch
from transformers import AutoModelForCTC, AutoProcessor

model_id = "dys-asr/parakeet-ctc-0.6b-soup"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForCTC.from_pretrained(model_id).eval()

inputs = processor(audio, sampling_rate=16_000, return_tensors="pt")  # 16 kHz mono
with torch.inference_mode():
    logits = model(**inputs).logits
print(processor.batch_decode(logits.argmax(dim=-1)))

Limitations

• —The 0.19-point gain is small and was selected on the reported set. See above.
• —No fair comparison against the sibling models on their own test sets, since one ingredient trained on SAPC1 dev.
• —The held-out set excludes Parkinson's speech entirely.
• —Numerals are written as words; no punctuation or casing.
• —Accuracy is very uneven across speakers, 0.7% to 70.9% WER.
• —English only, 16 kHz mono.
• —Single seed per ingredient. No variance estimate.