C10X/simplestories-bpe-2048

SimpleStories BPE 2048 Tokenizer

This is a 2,048-token byte-level BPE tokenizer trained on the complete training split of `SimpleStories/SimpleStories`.

Training details

• —Dataset revision: e63b8adc3b1a1bdc7cac5b500d150b71346b0628
• —Training examples: 2,115,696 stories
• —Text column: story
• —Pre-tokenizer and decoder: byte level, without an added prefix space
• —Vocabulary size: 2,048, including three special tokens
• —Model maximum length: 1,024 tokens
• —Normalization: none; capitalization and whitespace are preserved
• —UTF-8 input is represented without unknown-token fallbacks

<unk> is also used for padding. This matches the effective special-token mapping exposed by AutoTokenizer.from_pretrained("C10X/tenny").

Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("C10X/simplestories-bpe-2048")
encoding = tokenizer("Once upon a time, a little fox found a blue key.")
print(encoding.tokens())

plain = tokenizer(
    "Once upon a time, a little fox found a blue key.",
    add_special_tokens=False,
)
print(tokenizer.decode(plain.input_ids))

The tokenizer works with AutoTokenizer, batched padding and truncation, PyTorch tensor output, and Hugging Face language-modeling data collators. add_special_tokens=True prepends <|start_story|>, matching C10X/tenny. The end-of-story token is registered but is not appended automatically; training pipelines can append it according to their sequence-packing policy.

The package is tested with Transformers 4.31 / Tokenizers 0.13.3 and with Transformers 5.16 / Tokenizers 0.23.1.

Plain vocabulary exports are included as bpe_2048.txt and basic_vocab.txt.