nragrawal/marathi-tokenizer-era

Marathi Tokenizer Demo

A web interface for the Marathi BPE tokenizer. This demo allows you to:

• —Enter Marathi text
• —See the encoded tokens
• —Verify the decoding
• —Check token counts

Usage

Simply enter Marathi text in the input box and see the tokenization results!