HSinghHuggingFace/hindi-bpe-tokenizer

Hindi BPE Tokenizer

A Streamlit web application for encoding Hindi text to BPE tokens and decoding tokens back to text.

Features

• —Encode Hindi text to BPE tokens and token IDs
• —Decode token IDs back to Hindi text
• —Pre-trained on 5,000,000 lines of Hindi text
• —Vocabulary size: 4,500 tokens
• —Includes special tokens: <pad>, <unk>, <s>, </s>

Usage

1. 1.Encoding: Enter Hindi text in the left panel and click "Encode"
2. 2.Decoding: Enter comma-separated token IDs in the right panel and click "Decode"

Technical Details

• —BPE (Byte Pair Encoding) tokenizer
• —Trained on IndicCorp Hindi dataset
• —Compression ratio > 3.2
• —Preserves Hindi Unicode range (\\u0900-\\u097F)