Apppublic

dispatchAI/tokenizer-visualizer

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face

0likes

README.md23 linesDownload Raw Back to root

1---2title: Tokenizer Visualizer3emoji: 🔤4colorFrom: blue5colorTo: purple6sdk: gradio7sdk_version: "4.44.0"8app_file: app.py9pinned: false10license: apache-2.011tags: [tokenizer, visualization, mcp, dispatchai]12---13 14# 🔤 dispatchAI Tokenizer Visualizer15 16**MCP Server** — visualize how tokenizers split text.17 18## Tools19- `tokenize_text(text, method)` — Tokenize with chosen method20- `compare_methods(text)` — Compare token counts across methods21 22🚀 [dispatchAI](https://huggingface.co/dispatchAI)23