Mnttnt21/tokenizer-bench

Tokenizer bench (Asolytics meta editor)

The block above is Hugging Face Spaces front-matter (it configures the Space). It renders as a small table on GitHub — harmless. See DEPLOY.md.

Side-by-side comparison of PHP word-segmentation engines for languages that don't separate words with spaces, plus Korean. Goal: pick the engine whose output best matches how the stores index keywords.

Languages & why

The key test case is 密码 ("password"): a good tokenizer keeps it as one word, while per-character splitting gives the misleading 密(secret) + 码(code) = "secret code".

Engines compared

• —ICU / intl — universal baseline, covers zh/ja/th/ko in one API.
• —Jieba (cut / search) — Chinese. search mode emits overlapping sub-grams (mimics store indexing).
• —MeCab — Japanese gold standard (mecab-ipadic), called via the mecab CLI.
• —Sudachi (A) — Japanese alternative (sudachipy), split mode A = shortest units; a second opinion against MeCab. Python sidecar (bin/sudachi_tokenize.py).
• —PyThaiNLP (newmm) — Thai dictionary segmentation (~62k words), via a Python sidecar (bin/thai_tokenize.py). Keeps Thai compounds whole where ICU over-splits.
• —PyThaiNLP (+ASO dict) — same engine plus fixtures/thai_userdict.txt, a list of ASO loanwords/brands (แอป, คริป, แคปคัท…) that patch newmm's out-of-vocabulary splits.
• —Kiwi — Korean morphological analyzer (kiwipiepy). The only engine that actually splits glued Korean keywords (영상편집 → 영상·편집); may over-split brand names (캡컷 → 캡·컷).
• —N-gram (bigram) — not a word segmenter: overlapping 2-grams for zh/ja, approximating how Apple indexes CJK (substring n-grams, not words). The "what the store probably does" baseline.
• —Space split — Korean baseline (Korean uses spaces).
• —MeCab-ko — optional extra Korean morphology, appears only if mecab-ko-dic is present.

Heavy Python engines (Kiwi, PyThaiNLP, Sudachi) implement a batch path so the per-keyword table renders with one subprocess per engine instead of one per keyword (src/BatchTokenizer.php). Each render still pays a one-time model-load cost per engine (~3–4s for Kiwi with the cong-global model; Kiwi is also handed the whole keyword list at once because its per-call Python↔C overhead is ~150ms). The progress overlay covers this. zh pages spend ~3s loading Jieba's big dictionary into PHP. For heavy concurrent use a persistent model daemon would be the next step, but per-render spawning keeps the bench simple.

Run

docker compose up --build
# open http://localhost:8080

Pick a language, paste a keyword list (one per line), compare columns.

Two view modes

• —Per keyword (default) — a table: one row per keyword, one column per engine, showing how that keyword splits into words. This is the real ASO use case (you paste a keyword list).
• —Whole text — treats the whole input as one string, with an engine card per column.

Each language ships a default keyword list (video-editing theme; Thai & Korean supplied by the user) so you see results immediately. The input, language, mode, and translate toggle are saved to `localStorage`, so your text survives reloads and is remembered per language.

English glosses

With the 🇬🇧 toggle on, each keyword (and each token, on hover in table mode) shows its English translation. This makes mis-segmentation obvious (e.g. 免费手机游戏 → 免waived 费fee 手hand … vs the real free / cell phone / game).

Translation uses Google's unofficial gtx endpoint straight from PHP (src/Translator.php) — no API key, no Node dependency. It's rate-limited / ToS-gray (fine for a local bench) and results are file-cached in the system temp dir, so repeated tokens cost nothing. Fresh network calls are budgeted per request (cached items are free); if the budget is hit, remaining glosses show — and a notice appears — reload to fill the next batch.

Layout

• —src/ — TokenizerInterface + one class per engine + EngineRegistry + Translator
• —public/index.php — web UI (modes, defaults, localStorage)
• —fixtures/defaults.php — default keyword list per language