SlayerLab/tokenizers

1---2license: mit3pretty_name: SlayerLab Tokenizers4configs:5- config_name: default6  data_files:7  - split: train8    path: data/train-*.parquet9- config_name: benchmark10  data_files:11  - split: test12    path: results/tokenizer_benchmark.parquet13- config_name: evidence_review14  data_files:15  - split: review16    path: results/author_evidence_scores.csv17- config_name: leaderboard18  data_files:19  - split: test20    path: results/provisional_leaderboard.parquet21---22 23# SlayerLab Tokenizers24 25Normalized tokenizer artifacts collected from the contributor directories in26[`slayerlabs/tokenizer`](https://github.com/slayerlabs/tokenizer/tree/1a5cd2c2e4df2287b4c19b3dbf5051f5d460fdc1),27pinned to source commit `1a5cd2c2e4df2287b4c19b3dbf5051f5d460fdc1`.28 29The dataset contains one row per tokenizer: the 38 workshop submissions plus30the canonical SlayerLab Polish 32k tokenizer by `kacperwikiel`. Use the Dataset31Viewer to sort, filter, and compare tokenizers without navigating folders.32 33## Columns34 35- `author`: contributor's exact GitHub username, resolved from the source repository's pull-request history.36- `size`: tokenizer vocabulary size.37- `name`: original filename.38- `quick_status`: whether the artifact is ready to load or needs custom conversion.39- `hf_loadable`: whether it loads directly with `tokenizers.Tokenizer.from_file(...)`.40- `format`: normalized artifact-format classification.41- `model_type`: tokenizer model type reported by the source.42- `merge_count`: number of BPE merge rules when available.43- `normalizer`, `pre_tokenizer`, `decoder`: quick configuration summary.44- `unk_token`, `added_tokens_count`: special-token readiness indicators.45- `reported_metrics`: evaluation numbers reported by the author, encoded as JSON.46- `metadata`: normalized metadata encoded as a JSON string.47- `tokenizer_json`: complete, lossless original JSON file contents.48- `source_repo`: repository containing the original artifact.49- `source_path`: original path in the source repository.50- `source_commit`: pinned source Git commit.51- `bytes`: original JSON file size.52- `sha256`: checksum of the original JSON file.53 54## Formats55 56- `hf_tokenizers` (13 files): loadable with `tokenizers.Tokenizer.from_file(...)`.57- `custom_bpe` (17 files): custom BPE artifacts with `meta` and `model` fields.58- `custom_vocab` (1 file): vocabulary/merge mapping rather than a full tokenizer runtime file.59- `custom_experiment` (7 files): custom experiment or result JSON containing tokenizer data and metrics.60 61All source files were parsed as valid JSON. A file being valid JSON does not62imply it implements the Hugging Face Tokenizers serialization schema; check the63`format` column before loading `tokenizer_json`.64 65Reported metrics were produced with different texts and procedures, so they are66useful for inspecting an individual submission but not for ranking authors. A67fair quality ranking requires running every compatible tokenizer against the68same held-out Polish evaluation corpus.69 70## Common diagnostic benchmark71 72The `benchmark` configuration evaluates all 39 artifacts on the same versioned,73ten-domain Polish diagnostic suite. Fourteen run natively through the Rust74`tokenizers` library and 25 run through the included custom-format adapters.75 76- 39/39 artifacts executed successfully.77- 32 are reconstructed exactly; 7 are `core_only` because their intended78  pre-tokenizer was not serialized.79- 37/39 preserve exact input; two normalized `KateMajzel` variants convert a80  tab to a space.81- No unknown tokens were observed in this suite.82 83The benchmark reports compression, round-trip behavior, unknown-token rate,84per-domain results, serialized size, and local throughput. Throughput is only85comparable within the same `runtime`; Python reference adapters must not be86speed-ranked against native Rust tokenizers.87 88This is a small synthetic diagnostic corpus, not a statistically representative89held-out benchmark and not evidence of downstream language-model quality. Raw90compression must be compared within vocabulary-size bands.91 92## Evidence review93 94The `evidence_review` configuration separately scores artifact usability,95documentation, evaluation protocol, reproducibility, and claims discipline.96These scores judge the submitted evidence package—not tokenizer performance—and97must not be combined with compression metrics into a single winner score. See98`EVIDENCE_REVIEW.md` for the full evidence and limitations.99 100## Provisional scoring leaderboard101 102The `leaderboard` configuration provides a transparent 0–100 diagnostic quality103score. Eligibility requires successful execution, exact adapter fidelity, exact104round-trip on every suite record, and zero observed unknown tokens. Seven105`core_only` reconstructions and two round-trip failures remain visible but are106unranked.107 108The score adjusts compression for vocabulary size using author-balanced fits in109each domain, then combines 80% mean domain percentile with 20% lower-quartile110domain percentile. This rewards balanced performance and prevents an author111with many variants from defining the baseline. Scores within two points share112a rank tier.113 114Artifact readiness, evidence-package quality, traceability, file size, and speed115are displayed separately and do not influence the quality rank. `kacperwikiel`116is marked as a reference baseline. Best-of-many rows are explicitly labeled for117selection bias. See `LEADERBOARD_METHODOLOGY.md` for the complete formula and118limitations.119