sanjuhs/ternary-tinystories-4096

TinyStories 4K-BPE token stream This is a deterministic, tokenized derivative used by the Ternary LLM Experiment. The source is roneneldan/TinyStories. tokenizer: byte-level BPE, vocabulary 4,096 token dtype: little-endian uint16 training stories: 2,119,719 training tokens: 488,174,163 validation stories: 21,990 validation tokens: 4,907,807 Each story is encoded with explicit beginning/end markers and concatenated into train.bin or validation.bin. metadata.json records the… See the full description on the dataset page: https://huggingface.co/datasets/sanjuhs/ternary-tinystories-4096.