SlayerLab/tokenizers

1# Polish tokenizer diagnostic suite2 3`polish_suite.jsonl` is a fixed, synthetic stress suite written for this4repository on 2026-08-26. It has not been copied from a corpus and makes no5claim of being held out from tokenizer training data. Each domain contains two6short examples so domain metrics are useful for inspection, not statistical7inference.8 9Run the benchmark with:10 11```bash12python3 benchmark_tokenizers.py13```14 15All dataset rows are retained. Native Hugging Face artifacts have16`adapter_status=native`; other formats use the local reference adapter. An17`adapter_fidelity` of `exact` means the serialized artifact contains enough18information to reproduce its behavior. `core_only` means the byte-BPE core is19lossless but intended pre-tokenization is missing, so its segmentation metrics20must not be ranked as if they represented the author's full tokenizer.21 22Metrics use Unicode code points for `chars`, UTF-8 bytes for `bytes`, and23Python Unicode `\\w+` spans for words. Special tokens are disabled during24encoding and retained during decoding. Throughput is the median of repeated25whole-suite batches after one warm-up and is only comparable within the same26run, machine, and `runtime`. In particular, `python_reference_adapter` speed27measures this diagnostic adapter and is not comparable to `rust_tokenizers`.28 29This suite can catch broken round trips, unknown-token behavior, pathological30segmentation, and operational cost differences. It cannot establish model31quality. A defensible final ranking also needs a larger provenance-controlled32held-out corpus and identical downstream language-model experiments.33