kurdish-tech/kurdish-tokenizer-unigram-32k

Kurdish Tokenizer (Unigram, 32k) — Kurmancî · Soranî · Zazakî

A Unigram (SentencePiece-style) tokenizer covering all three major Kurdish varieties in a single vocabulary: Kurmancî (Latin script), Soranî (Arabic script), and Zazakî (Latin script).

Trained by Kurdish-Tech on KurdishCorpus-clean, and measured on held-out text the tokenizer never saw during training.

Which variant should I use?

Honestly, probably not this one. Four Kurdish tokenizers were trained and benchmarked together, and this 32k-vocabulary Unigram variant has the second-weakest fertility of the four. Its only reason to exist is a smaller embedding table (32k vs 64k rows) for parameter-constrained models. If that constraint doesn't apply to you, use kurdish-tokenizer-unigram-64k instead — same algorithm, the best Soranî/Zazakî fertility of all four variants.

Fertility (tokens per word — lower is better)

Measured on held-out documents: 300 per dialect (166 for Zazakî — all that met the threshold), each ≥20 words, truncated to 2,000 characters. Every tokenizer below was measured with the same script on the same documents, encoding without special tokens.

Still beats cl100k_base on every dialect (1.8× fewer tokens on Kurmancî, 3.8× fewer on Soranî) despite being the second-weakest of the four Kurdish variants.

Usage

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("kurdish-tech/kurdish-tokenizer-unigram-32k")

tok("Ez kurd im, ji Kurdistanê me.")          # Kurmancî
tok("زمانی کوردی زمانێکی دەوڵەمەندە.")          # Soranî
tok("Ma kirmanc î, zon u kulturê ma.")        # Zazakî

Other variants

• —kurdish-tokenizer-unigram-64k — same algorithm, best Soranî/Zazakî fertility of all four (recommended)
• —kurdish-tokenizer-bpe-64k — best Kurmancî fertility, standard choice for causal LM
• —kurdish-tokenizer-bpe-32k — BPE at the same vocab size

Limitations

• —Fertility is a relative comparison on a held-out sample, not an exhaustive evaluation.
• —Published for completeness and for use cases genuinely constrained on vocabulary size, not as the general recommendation.
• —This is a tokenizer only — no language model weights are released here.

License & citation

Released under CC BY-SA 4.0, matching the corpus it was trained on.

@misc{kurdishtech2026tokenizer,
  title  = {Kurdish Tokenizer (Unigram 32k): a multi-dialect tokenizer for Kurmanc\^i, Soran\^i and Zazak\^i},
  author = {{Kurdish-Tech}},
  year   = {2026},
  url    = {https://huggingface.co/kurdish-tech/kurdish-tokenizer-unigram-32k}
}

Built by [Kurdish-Tech](https://github.com/Kurdish-Tech) — open-source digital infrastructure for the Kurdish language.

Maintained by Alan Hesen.