billingsmoore/tibetan-word-segmentation-ds

Tibetan Word Segmentation Annotations

Human expert word boundary annotations for 100 utterances of modern Lhasa Tibetan news speech, drawn from the NICT-Tib1 ASR corpus.

Dataset Description

Tibetan orthography does not mark spaces between words; syllables are delimited only by the tsek (་) character. This dataset provides human-annotated word segmentations that serve as the reference standard in a five-way segmentation agreement study comparing automatic Tibetan word segmenters against human judgment.

These annotations were created for and used in the segmentation validation study reported in:

J. Moore, S. Li and P. Lauren, "Evaluating Tibetan ASR With Segmented Word Error Rate: Beyond Character-Level Metrics," in IEEE Access, vol. 14, pp. 101790-101805, 2026, doi: 10.1109/ACCESS.2026.3709206.

Annotation Process

Sampling. 100 utterances were randomly sampled from the 1,547-utterance NICT-Tib1 test split, stratified by speaker in proportion to the speaker distribution of the full test set (speakers 019, 070, and 074, the three speakers held out for testing — see billingsmoore/nict-tib1). This yielded 2,634 inter-syllable boundary positions.

Annotation. A single human expert with Tibetan language expertise (Paul Hackett) annotated word boundaries for all 100 utterances independently, working from the reference transcriptions only. As the paper notes, a single annotator's judgments are not a consensus gold standard — the reported agreement statistics measure agreement with one individual's segmentation standard, and inter-annotator agreement was not separately assessed.

Agreement study. The same 100 utterances were independently segmented by four automatic systems — Botok (dictionary-based), Gemini 2.5 Flash Lite, BERT-UPOS, and Claude Haiku (claude-haiku-4-5, included as an additional LLM reference point but not used to compute a SWER variant in the paper's main evaluation) — with no system seeing the gold boundaries or any other system's output. Boundary-level agreement with the human annotation was measured using Cohen's κ and boundary F1 (human annotation as ground truth):

Gemini achieved substantial agreement with the human annotator (Landis–Koch scale), establishing it as the most human-aligned automatic segmenter evaluated. Botok's high boundary rate — a consequence of its out-of-vocabulary fallback on this news-domain corpus — made it a consistent outlier relative to every other system, including the human annotator.

Dataset Structure

Example

Source Data

Audio and reference transcriptions are from the NICT-Tib1 corpus (Soky, Gong & Li, 2022), a public speech corpus of Lhasa dialect Tibetan for benchmarking ASR systems. The 100 utterances here are a stratified sample of the 1,547-utterance test split used in the associated paper; NICT-Tib1 as a whole contains 20 speakers, split 85/15 by speaker (17 training / 3 test) so no speaker appears in both splits.

Intended Uses

• —Evaluating and benchmarking Tibetan word segmentation systems
• —Computing segmentation-dependent ASR metrics (e.g., Word Error Rate) for Tibetan
• —Training or fine-tuning segmentation models on modern Lhasa Tibetan news speech

Limitations

• —Annotations cover only modern Lhasa (Central) Tibetan; they may not reflect boundary conventions for Amdo or Kham dialects, or Classical Tibetan literary text.
• —100 utterances is a small sample; boundary decisions on genuinely ambiguous positions reflect one annotator's judgment, and inter-annotator agreement was not assessed — this is not a consensus gold standard.
• —The source corpus is read news speech; segmentation conventions may differ in spontaneous or conversational domains.

Acknowledgments

Word boundary annotations were provided by Paul Hackett, whose contribution is acknowledged in the associated paper.

Related Resources

• —tibetan-wer (PyPI): open-source package implementing CER, SER, and Segmented WER for Tibetan ASR evaluation — https://pypi.org/project/tibetan-wer/
• —NICT-Tib1 ASR dataset: billingsmoore/nict-tib1
• —Model predictions: billingsmoore/preds

Citation

If you use this dataset, please cite both the paper and the source NICT-Tib1 corpus:

@ARTICLE{11592371,
  author={Moore, Jacob and Li, Sheng and Lauren, Paula},
  journal={IEEE Access},
  title={Evaluating Tibetan ASR With Segmented Word Error Rate: Beyond Character-Level Metrics},
  year={2026},
  volume={14},
  number={},
  pages={101790-101805},
  keywords={Modeling;Automatic speech recognition;Error analysis;LoRa;Measurement;Ranking (statistics);Quantization (signal);Bit error rate;Standards;Training;Tibetan;automatic speech recognition;word error rate;low-resource language},
  doi={10.1109/ACCESS.2026.3709206}
}

@inproceedings{soky2022nict,
  title={Nict-tib1: A public speech corpus of lhasa dialect for benchmarking tibetan language speech recognition systems},
  author={Soky, Kak and Gong, Zhuo and Li, Sheng},
  booktitle={2022 25th Conference of the Oriental COCOSDA International Committee for the Co-ordination and Standardisation of Speech Databases and Assessment Techniques (O-COCOSDA)},
  pages={1--5},
  year={2022},
  organization={IEEE}
}