Apppublic

smcproject/malayalam-tokenizer-comparison

sourceHugging Facemitupdated 7mo agoView on Hugging Face

2likes

App README

Malayalam Tokenizer Comparison

Visualize how different tokenizers split Malayalam text into tokens. Compare Malayalam-specific tokenizers (BPE and Unigram) side-by-side with popular models like GPT-4, LLaMA, Mistral, and others.

Built from smc/malayalam-tokenizer.