Richard-Sieg-TH-Koln/anlp-tinystories-gpt2
TinyStoriesV2-GPT4, tokenized for GPT-2 Pre-tokenized version of the TinyStoriesV2-GPT4 split of roneneldan/TinyStories, prepared for the Advanced NLP block course at TH Koeln. Tokenizer: tiktoken, GPT-2 encoding (tiktoken.get_encoding("gpt2")) Format: flat uint16 array per split, <|endoftext|> (token id 50256) between stories Files: train.bin, valid.bin Read with: import numpy as np tokens = np.memmap("train.bin", dtype=np.uint16, mode="r") Original stories by Eldan and Li… See the full description on the dataset page: https://huggingface.co/datasets/Richard-Sieg-TH-Koln/anlp-tinystories-gpt2.
Conversations for this repository live on Hugging Face.
CoolFace shows imported repositories read-only. Posting into someone else’s repository from here would need an authorised integration and the account holder’s consent, so the link goes to the source instead.
Open discussions on Hugging Face