Richard-Sieg-TH-Koln/anlp-tinystories-gpt2

TinyStoriesV2-GPT4, tokenized for GPT-2 Pre-tokenized version of the TinyStoriesV2-GPT4 split of roneneldan/TinyStories, prepared for the Advanced NLP block course at TH Koeln. Tokenizer: tiktoken, GPT-2 encoding (tiktoken.get_encoding("gpt2")) Format: flat uint16 array per split, <|endoftext|> (token id 50256) between stories Files: train.bin, valid.bin Read with: import numpy as np tokens = np.memmap("train.bin", dtype=np.uint16, mode="r") Original stories by Eldan and Li… See the full description on the dataset page: https://huggingface.co/datasets/Richard-Sieg-TH-Koln/anlp-tinystories-gpt2.