Richard-Sieg-TH-Koln/anlp-tinystories-gpt2

TinyStoriesV2-GPT4, tokenized for GPT-2

Pre-tokenized version of the TinyStoriesV2-GPT4 split of roneneldan/TinyStories, prepared for the Advanced NLP block course at TH Koeln.

• —Tokenizer: tiktoken, GPT-2 encoding (tiktoken.get_encoding("gpt2"))
• —Format: flat uint16 array per split, <|endoftext|> (token id 50256) between stories
• —Files: train.bin, valid.bin

Read with:

import numpy as np
tokens = np.memmap("train.bin", dtype=np.uint16, mode="r")

Original stories by Eldan and Li, 2023, TinyStories: How Small Can Language Models Be and Still Speak Coherent English?. This repository contains only a tokenized derivative of the GPT-4-generated (v2) split, released under the original dataset's CDLA-Sharing-1.0 license.

Note on v1 vs v2

The source files were downloaded from TinyStoriesV2-GPT4-train.txt and TinyStoriesV2-GPT4-valid.txt in the upstream repo, confirmed by an exact byte-count match against the hub. TinyStoriesV2 is a cleaned-up superset of v1, not a full regeneration: some v1 stories carry over into v2 verbatim, including a few early ones in this file with the more GPT-3.5-flavored, mildly surreal tone (confirmed by searching the actual v1 file for the same text). So spotting a stylistically odd early story is not a sign that this is secretly v1, that is expected v2 behavior.