CoolFace
Datasetpublic

Exqrch/Rebuttal-javanese-pixelgpt

Javanese PixelGPT Tokenizer Ablation Dataset Optimized with Font Size 6 and Dynamic Trimming. Tokenizer Schema tok_grapheme: Language-specific Grapheme BPE (izzako/javanese-llama-tokenizer) tok_llama2: Standard Llama-2 BPE (ernie-research/DualGPT) tok_komodo: SEA-Optimized BPE (yellow-ai-central/komodo-7b-v1) tok_mt5: Google Multilingual Unigram (google/mt5-small)

sourceHugging Faceupdated 6mo agoView on Hugging Face
0likes89downloads
Dataset Card

Javanese PixelGPT Tokenizer Ablation Dataset

Optimized with Font Size 6 and Dynamic Trimming.

Tokenizer Schema

  1. 1.tok_grapheme: Language-specific Grapheme BPE (izzako/javanese-llama-tokenizer)
  2. 2.tok_llama2: Standard Llama-2 BPE (ernie-research/DualGPT)
  3. 3.tok_komodo: SEA-Optimized BPE (yellow-ai-central/komodo-7b-v1)
  4. 4.tok_mt5: Google Multilingual Unigram (google/mt5-small)