speakleash/bielik-tokenizers

Tokenizer Playground

Inspect how text is tokenized with any Hugging Face tokenizer. Use it locally or on Hugging Face Spaces.

Features

• —Input text — Main textarea to type or paste text.
• —Tokenization — See each token with its ID in the form token (id).
• —Stats — Character count, token count, chars per token.
• —Tokenizer config — EOS, PAD, BOS, UNK, SEP, CLS tokens and IDs; model_max_length, vocab_size.
• —Chat template — Raw template string and an example with apply_chat_template.

Usage

1. 1.Enter a model / tokenizer ID (e.g. gpt2, meta-llama/Llama-2-7b-hf, Qwen/Qwen2.5-0.5B).
2. 2.Optionally click Load tokenizer & refresh config to load the tokenizer and see config + chat template.
3. 3.Type or paste text in Input text and click Tokenize to see tokens, IDs, and stats.

Running locally

No PyTorch/Torch is required — the app uses only the tokenizer and config, so installs stay small.

pip install -r requirements.txt
python app.py

Deploying on Hugging Face Spaces

Push this repo to a Space with Gradio SDK. The README frontmatter above configures the Space; app.py and requirements.txt are used automatically.