dys-asr/parakeet-rnnt-0.6b-all-syn-chunk-cutout-bpedrop

Parakeet RNN-T 0.6B — BPE-dropout

The ten-epoch recipe behind `dys-asr/parakeet-rnnt-0.6b-all-syn-chunk-cutout`, rerun with BPE-dropout at p = 0.1 on the training targets and nothing else changed. Each merge is skipped with that probability when a transcript is tokenised, so the same sentence is segmented differently from epoch to epoch and the prediction network never settles on one spelling of a word.

What it is worth

On the held-out dev split, against the run it varies:

Treat both columns with suspicion. The subset and the full split disagree about the ordering across this family of runs, the spread is under 0.2 CER points, and there is one seed behind each row. The full split is 17,492 utterances against the subset's 4,000 and is the more trustworthy of the two, but neither is a significance test.

Every run in this family peaked at epoch 9 or 10 with no sign of overfitting, so ten epochs may simply be short.

What had to change to make it work

ParakeetProcessor.__call__ tokenises the transcript twice — once for labels, and again for decoder_input_ids after prepending the blank token. With sampling on, those two calls draw different segmentations of the same sentence, the joint's U+1 stops matching max(target_lengths) + 1, and torchaudio raises output length mismatch on the first step. The training collator here tokenises once and builds the decoder input from that single sample. With dropout off the result is bit-identical to what the processor produces, which is what keeps the other arms comparable.

Sampling also lengthens targets: an 18-token sentence measured between 18 and 22 tokens on this tokeniser. The evaluation tokeniser is a separate object and is never given a dropout, so dev scores do not move with the sampling.

Training data

The same corpus as `dys-asr/parakeet-rnnt-0.6b-all-syn-chunk-cutout`, unchanged, so the two are directly comparable.

463,177 records survive the 0.5-45 s and 200-label-token filters. Nothing comes from outside the challenge corpora, so this is a constrained-track model.

The held-out dev split is genuinely held out. sapc2_dev_heldout.jsonl is 17,582 clips and its intersection with the training manifests is empty; that was checked by comparing file paths, not assumed.

Training procedure

Ten epochs on sixteen GH200s, effective batch 32 at per-device 2, AdamW at 1e-4 with a tri-stage schedule (10% warmup, 40% hold), weight decay 0.01, layerdrop 0.05, gradient clip 1.0, bf16, seed 42, 144,750 optimizer steps. Augmentation is online speed perturbation over 0.8-1.2, SpecAugment (5% of the time axis in spans of 10 frames, 40% of the mel axis in spans of 27 bins) and SpecCutout (two 20x20 rectangles).

Usage

import soundfile as sf
import torch
from transformers import AutoProcessor, ParakeetForRNNT

model_id = "dys-asr/parakeet-rnnt-0.6b-all-syn-chunk-cutout-bpedrop"
device = "cuda" if torch.cuda.is_available() else "cpu"

processor = AutoProcessor.from_pretrained(model_id)
model = ParakeetForRNNT.from_pretrained(model_id).to(device).eval()

audio, rate = sf.read("utterance.wav", dtype="float32")
inputs = processor(audio, sampling_rate=rate, return_tensors="pt").to(device)
with torch.inference_mode():
    generated = model.generate(**inputs)
print(processor.batch_decode(generated, skip_special_tokens=True)[0])

Requires transformers>=5.9. Audio must be 16 kHz mono.

Intended use and limitations

A competition entry for the Speech Accessibility Project Challenge, and research on atypical-speech recognition.

• —Single run, single seed. No variance estimate. The differences reported above are smaller than a seed sweep would likely show, and should not be read as a ranking of methods.
• —Requires `transformers>=5.9`. Not loadable by the 4.x line.
• —Slow relative to CTC. Decoding is autoregressive.
• —Lower-case, unpunctuated, numerals written as words.
• —English only, 16 kHz mono.
• —Not a clinical tool. Nothing here supports inference about any diagnosis.

License and attribution

The Speech Accessibility Project corpora are governed by their own data use agreement and are not redistributed here; reproducing this training set requires authorized access. Fun-CosyVoice3's terms apply to the synthesis component.