CoolFace
Modelpublic

AksaraLLM/aksara-tokenizer-v1

sourceHugging Faceapache-2.0updated 6mo agoView on Hugging Face
0likes
Model Card

AksaraLLM Tokenizer v1

Custom BPE tokenizer optimized for Indonesian and local languages.

Stats

  • —Vocab Size: 32,768
  • —Algorithm: Byte-Pair Encoding (BPE)
  • —Pre-tokenizer: ByteLevel
  • —Training Data: AksaraLLM pre-train + SFT corpus

Supported Languages

  • —Bahasa Indonesia (ID)
  • —Bahasa Jawa (JV)
  • —Bahasa Sunda (SU)
  • —Bahasa Bali (BAL)
  • —Bahasa Batak (BTK)
  • —Bahasa Bugis (BUG)
  • —Bahasa Minangkabau (MIN)
  • —Bahasa Madura (MAD)
  • —Bahasa Aceh (ACE)
  • —Bahasa Banjar (BJN)
  • —English (EN)

Special Tokens (29)

IDTokenPurpose
0[PAD]Padding
1[EOS]End of sequence
2[BOS]Begin of sequence
3[UNK]Unknown
4[SEP]Separator
5[MASK]Mask
6[SYSTEM]System prompt
7[USER]User message
8[ASST]Assistant message
9[INST]Instruction start
10[/INST]Instruction end
11-21[LANG_*]Language markers
22[TURN]Turn separator
23-24[THINK]/[/THINK]Chain-of-thought
25-26[CODE]/[/CODE]Code blocks

Usage

python
from tokenizers import Tokenizer

# Load
tok = Tokenizer.from_file("tokenizer.json")
# or from HuggingFace:
# from huggingface_hub import hf_hub_download
# path = hf_hub_download("AksaraLLM/aksara-tokenizer-v1", "tokenizer.json")
# tok = Tokenizer.from_file(path)

# Encode
encoded = tok.encode("Selamat pagi, apa kabar?")
print(encoded.ids)
print(encoded.tokens)

# Decode
decoded = tok.decode(encoded.ids)
print(decoded)

Comparison vs GPT-2

TextGPT-2AksaraLLMSaving
"Selamat pagi"3-5 tokens2 tokens~50%
"kemerdekaan"3-4 tokens1-2 tokens~60%
"Pancasila"3-4 tokens1 token~70%

Fewer tokens = faster inference + cheaper training + better quality.

License

Apache 2.0