wayu-ai/thai-aligner-bench

Thai Aligner Bench

🚧 Development in progress.

How accurately can a forced aligner place Thai token and word boundaries in speech? This is a self-contained benchmark: one Python file (aligner_bench.py) plus 1,572 clips of Thai speech with frame-exact timing ground truth. No Thai NLP stack or other code is needed — just numpy soundfile torch torchaudio transformers.

The ground truth is what makes the dataset useful: the audio was rendered by a TTS model whose duration predictor emitted an explicit frame count per token (pred_dur, 25 ms per frame; the frame counts sum exactly to the audio length). The token boundaries in the audio therefore are the pred_dur boundaries — no human annotation, no second aligner, no circularity. Any forced aligner can be measured against them directly.

Each aligner is scored on timing accuracy: onset error percentiles, signed bias, jitter (spread with bias removed), duration error, and span coverage, all measured against the pred_dur boundaries.

Quick start

git lfs install   # the audio is stored with Git LFS
git clone https://huggingface.co/datasets/wayu-ai/thai-aligner-bench-dev
cd thai-aligner-bench-dev
pip install numpy soundfile torch torchaudio transformers

python aligner_bench.py run --bundle . --aligner ctc --out out/ctc
python aligner_bench.py run --bundle . --aligner mms --out out/mms
python aligner_bench.py report out/ctc out/mms

run = align (GPU, writes spans.jsonl) + score (CPU, writes metrics.json). Run them separately to re-score without re-aligning.

Useful flags: --limit N / --speakers spk00,spk03 (fast subsets), --quantize-ms (round predicted onsets onto a coarser grid — tests whether an aligner's deficit is just its frame rate), --device, --batch-size.

Aligners included

Add your own aligner

Write one function returning [(char_idx, t0_ms, t1_ms)] over item["text"], time-monotonic, and register it:

def mine_bundle(device, **_):
    return SimpleNamespace(..., tag="mine:v1", batched=False)

def mine_char_spans(wav, sr, item, b):
    ...
    return spans

ALIGNERS["mine"] = (mine_bundle, mine_char_spans, None)

Everything downstream of the aligner — span collection, scoring, aggregation, reporting — is shared, so results stay comparable.

If your aligner batches clips, pass an explicit length mask. wav2vec2-large-xlsr-53-th ships with return_attention_mask: false even though it was trained with masking; running it unmasked silently stretches each clip's timeline — over a second of drift on longer clips. The bundled aligners already handle this.

Reading the metrics

• —`bias` is correctable, `jitter` is not. Aligners report the left edge of a frame, so each carries a systematic half-frame early bias that a constant shift removes. Jitter is what decides usability.
• —*Never compare CTC span widths across aligners.* They allocate blank frames differently, so coverage varies widely between models, and measuring duration as a span's own width can reverse a ranking. Duration error therefore uses onset[i+1] − onset[i], the convention pred_dur itself uses.
• —Separate model error from grid resolution. --quantize-ms re-scores an aligner with its onsets rounded onto a coarser grid; if the numbers barely move, the aligner's deficit is coming from the model, not its frame rate.

Dataset structure

items.jsonl        one JSON per clip (1,572)
audio/<id>.flac    24 kHz mono, lossless
aligner_bench.py   the benchmark (single file)

Limitations

• —The audio is vocoder output, not natural speech. It is out of domain for aligners trained on read speech. This is the price of the ground truth — frame-exact boundaries exist precisely because the audio was generated from them — but results here are not a claim about natural speech, and an aligner that does well on real speech may rank differently here.
• —*`pred_dur` is ground truth for this TTS model's rendering*, not a human phonetic transcription. It is exact about where the model placed each token; it does not adjudicate whether that placement was phonetically ideal.
• —Synthetic speakers. 12 designed voices from a single TTS model, so speaker and acoustic diversity is narrower than a natural-speech corpus.