kishkath/bpe-tokenizer

Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference

Telugu Text Tokenizer

A Gradio web interface for encoding and decoding Telugu text using a trained BPE tokenizer.

Features

• —Encode Telugu text to token IDs
• —View compression statistics and token visualization
• —Decode token IDs back to Telugu text
• —Interactive and user-friendly interface

Usage

1. 1.Encoding Text
2. 2.Enter Telugu text in the encoder tab
3. 3.Click "Encode" to get token IDs and statistics
4. 4.View token segmentation with color visualization

1. 1.Decoding Text
2. 2.Paste encoded token IDs in the decoder tab
3. 3.Click "Decode" to get back the original text

Technical Details

• —Uses Byte Pair Encoding (BPE) algorithm
• —Vocabulary size: 4800 tokens
• —Supports efficient compression of Telugu text
• —Maintains perfect reconstruction

Model Information

The tokenizer is trained on a diverse corpus of Telugu text with:

• —Maximum vocabulary size: 5000 tokens
• —Target compression ratio: ≥ 3.2x
• —Perfect reconstruction guarantee