ravis2k6/tamil_bpe_tokenizer

Tamil BPE Tokenizer

This is a Byte Pair Encoding (BPE) tokenizer optimized for Tamil text. The model achieves a compression ratio of 4.0 while maintaining a vocabulary size of 5000 tokens.

Features

• —Tamil-aware tokenization
• —Compression ratio: 4.0
• —Vocabulary size: 5000 tokens
• —Preserves Tamil linguistic structures

Usage

1. 1.Enter Tamil text in the input box
2. 2.Get tokenized output and statistics
3. 3.See compression ratio achieved

Model Details

• —Trained on Tamil corpus
• —Optimized for Tamil character combinations
• —Handles vowel markers appropriately