izzako/javanese-pixelgpt

Javanese PixelGPT Dataset

This dataset contains preprocessed Javanese text data for training PixelGPT models.

Dataset Statistics

• —Language: Javanese (jawa)
• —Total samples: 401,542
• —Train samples: 400,726
• —Test samples: 816

Tokenizers

• —Grapheme tokenizer: izzako/javanese-llama-tokenizer
• —LLaMA tokenizer: ernie-research/DualGPT

Features

• —text_id: Document identifier
• —chunk_id: Chunk identifier within document
• —pixel_values: Rendered pixel representation of aksara text
• —grapheme_token_ids: Token IDs from grapheme-based tokenizer (with EOS token)
• —llama_token_ids: Token IDs from LLaMA tokenizer (with EOS token)
• —text: Original text chunk

Renderer Configuration

• —Renderer path: ../../renderers/m4_renderer

Usage

from datasets import load_dataset

# Load the dataset
dataset = load_dataset("izzako/javanese-pixelgpt")

# Access train and test splits
train_data = dataset['train']
test_data = dataset['test']

# Example: Get first sample
sample = train_data[0]
print(sample['text'])

Citation

If you use this dataset, please cite:

@dataset{javanese_pixelgpt,
  title = {Javanese PixelGPT Dataset},
  author = {Musa Izzanardi Wijanarko},
  year = {2025},
  publisher = {Hugging Face},
  url = {https://huggingface.co/datasets/izzako/javanese-pixelgpt}
}