datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
TinyStoriesDataset containing synthetically generated (by GPT-3.5 and GPT-4) short stories that only use a small vocabulary.
Described in the following paper: https://arxiv.org/abs/2305.07759.
The models referred to in the paper were trained on TinyStories-train.txt (the file tinystories-valid.txt can be used for validation loss). These models can be found on Huggingface, at roneneldan/TinyStories-1M/3M/8M/28M/33M/1Layer-21M.
Additional resources:
tinystories_all_data.tar.gz - contains a superset of… See the full description on the dataset page: https://huggingface.co/datasets/roneneldan/TinyStories.TinyStories-Algerian-Darijav0-tinystories-v2-clean-tokenizedv0 refers to our tokenization process, v2 refers to stories generated by GPT4 only
TinyStoriesInstructTinyStoriesV2Data is from https://huggingface.co/datasets/roneneldan/TinyStories/:
TinyStoriesV2-GPT4-train.txt - Is a new version of the dataset that is based on generations by GPT-4 only (the original dataset also has generations by GPT-3.5 which are of lesser quality). It contains all the examples in TinyStories.txt which were GPT-4 generated as a subset (but is significantly larger).
This dataset was used to train https://github.com/noanabeshima/tiny_model/.
The data was preprocessed with:
from… See the full description on the dataset page: https://huggingface.co/datasets/noanabeshima/TinyStoriesV2.tinystories-gpt4-clean
TinyStories GPT-4 Clean
A cleaned subset of the TinyStories dataset (Eldan & Li, 2023), keeping only GPT-4-generated stories. Adapted from this thread that pointed out many issues with the original data and proposed a cleaning process.
Overview
This cleaned dataset contains:
Stat
Value
Stories
2,732,634
Total characters
~2.19B
Min doc length
115 chars
Max doc length
4,433 chars
Median doc length
721 chars
Unique characters
74 (ASCII only)
Duplicates… See the full description on the dataset page: https://huggingface.co/datasets/karpathy/tinystories-gpt4-clean.somali-tinystoriessae-skeskinen-TinyStories-hf-validation-tokenizer-gpt2_playtinystoriesTinyStoriesV2-Tokenizedtinystories-icr-data-v2TinyStoriesAdv-zh
TinyStoriesAdv
keywords: grade school level, large language model, small language model, tiny language model, super tiny language model, 小学生知识水平，大语言模型，小语言模型，迷你语言模型, llm, slm.
受到TinyStories、Phi2等论文的启发，我制作了一个约1B tokens的小学知识水平的“一揽子”大语言模型训练语料库。
“一揽子”指的是本数据集是众多数据集的集合。为了提升模型的不同能力（例如事实性知识、元认知、思维链、阅读理解RAG、逻辑推理等），我开了不少脑洞，使用了多种创新的提示词生成了具有多样性和针对性的子数据集。… See the full description on the dataset page: https://huggingface.co/datasets/fzmnm/TinyStoriesAdv-zh.TinyStories2-ascii-bpe-32k
Dataset Card for "TinyStories2-ascii-bpe-32k"
More Information needed
tinystories_dataset_arabicTinyStoriesV2_cleaned
License: CDLA-Sharing-1.0
Dataset containing synthetically generated (GPT-4) short stories that only use a small vocabulary.
Described in the following paper: https://arxiv.org/abs/2305.07759.
This is a cleaned up Version of the original TinyStories Dataset: https://huggingface.co/datasets/roneneldan/TinyStories.
We thank the authors for their contribution.
This Version only contains cleaned-up stories generated by GPT4.
Stories were deleted that contained spelling and… See the full description on the dataset page: https://huggingface.co/datasets/fhswf/TinyStoriesV2_cleaned.pa1-tinystories
CS 5326: Advanced Generative AI and Agents
Programming Assignment 1: The Modern Transformer LM
This dataset accompanies Programming Assignment 1 for CS 5326: Advanced
Generative AI and Agents. It gives every student the same ready-to-use text
corpus and tokenizer for implementing and training a modern Transformer language
model from scratch.
Source and credits
The text comes from roneneldan/TinyStories,
introduced by Ronen Eldan and Yuanzhi Li in… See the full description on the dataset page: https://huggingface.co/datasets/alooboii/pa1-tinystories.tinystories_phonologyTinyStories
Pretokenized TinyStories
Based on roneneldan/TinyStories
105 Tokens byte_fallback=False
128 Tokens byte_fallback=False
210 Tokens byte_fallback=False
361 Tokens
4k Tokens
32K Tokens
includes:
tok*.vocab
tok*.model
tok*.bin
tok*.tar.gz
data{00..49}.bin
Pretokenized to speed up training on:
karpathy/llama2.c
EN10/BabyLlama
TinyStories-gpt2-cache-100kCached activations at layer 5 for gpt2 using dataset apollo-research/roneneldan-TinyStories-tokenizer-gpt2
Useful for accelerated training and testing of sparse autoencoders
context_window: 512 tokens
total_tokens: 51,200,000
batch_size: 8 prompts (4096 tokens)
layer_hook_name: blocks.5.hook_mlp_out
tinystories-processed_512multilingual_TinyStories
Dataset Card for Multilingual TinyStories
Dataset Details
Dataset Description
The Multilingual TinyStories dataset contains translations of the original TinyStories dataset, which consists of synthetically generated short stories using a small vocabulary suitable for 3 to 4-year-olds. These stories were originally generated by GPT-3.5 and GPT-4. The multilingual versions have been translated into various languages, including Spanish, Chinese, German, Turkish… See the full description on the dataset page: https://huggingface.co/datasets/Gabrui/multilingual_TinyStories.flan-ul2-tinystoriesAround a quarter of a million examples generated from Flan-UL2 (20b) with the prompt "Write a short story using the vocabulary of a first-grader." to be used in an experimental curriculum learning setting. I had to checkpoint every 1024 examples to mitigate the program slowing down due to memory usage. This was run in bf16 on an RTXA6000 with the following settings:
top_k = random between (40, 128)
temperature = random between (0.6, 0.95)
max_length = 128
batch_size = 32
I wanted a less… See the full description on the dataset page: https://huggingface.co/datasets/crumb/flan-ul2-tinystories.multilingual_tinystoriesAn TinyStories dataset for Spanish. The code to generate this is here. https://github.com/rrenaud/multilingual_tinystories
flan-ul2-tinystories-complexAround a quarter of a million examples generated from Flan-UL2 (20b) with the prompt "Write a complex short story using the vocabulary of a third-grader." to be used in an experimental curriculum learning setting. I had to checkpoint every 1024 examples to mitigate the program slowing down due to memory usage. This was run in bf16 on an RTXA6000 with the following settings:
top_k = random between (40, 128)
temperature = random between (0.6, 0.95)
max_length = 128
batch_size = 32
I wanted a… See the full description on the dataset page: https://huggingface.co/datasets/crumb/flan-ul2-tinystories-complex.TinyStories-Multilingual
Novelist: TinyStories Multilingual Edition
Dataset Summary
The TinyStories Multilingual Edition is a high-fidelity synthetic dataset of short, child-safe fiction designed to stress-test literary consistency, emotional warmth, and multilingual fluency in small models. Derived from the broader Novelist ecosystem, this subset focuses on narrative simplicity paired with complex moral and social themes.
The dataset contains 15,688 high-quality stories across 28 languages. Each… See the full description on the dataset page: https://huggingface.co/datasets/Dxniz/TinyStories-Multilingual.TinyStories-GPT4
Dataset Card for "TinyStories-GPT4"
More Information needed
TinyStories2-ascii
Dataset Card for "TinyStories2-ascii"
TinyStoriesV2-GPT4-{train,validation}.txt from roneneldan/TinyStories
ad-hoc Unicode -> ASCII normalization
remove empty/incomplete stories
TinyStoriesZhLM朝着越来越大的方向卷，而在小LM的方向，有研究者在探索小LM方向的边界能力，比如想知道多小的语言模型仍然能流畅的说故事？
TinyStories 是在其做该方向时使用的一份关于小故事的场景数据。故事是由研究者使用GPT3.5, GPT4生成的，并且将故事难度限制在3~4岁小朋友能理解。
这份中文数据通过翻译器将英文故事数据翻译而成。如下例子。
Lily and Ben are friends. They like to play in the park. One day, they see a big tree with a swing. Lily wants to try the swing. She runs to the tree and climbs on the swing.\n"Push me, Ben!" she says. Ben pushes her gently. Lily feels happy. She swings higher and higher. She laughs and shouts.\nBen watches Lily. He… See the full description on the dataset page: https://huggingface.co/datasets/52AI/TinyStoriesZh.Tiny-Stories-tokenizedTinyStoriesDataset containing synthetically generated (by GPT-3.5 and GPT-4) short stories that only use a small vocabulary.
Described in the following paper: https://arxiv.org/abs/2305.07759.
The models referred to in the paper were trained on TinyStories-train.txt (the file tinystories-valid.txt can be used for validation loss). These models can be found on Huggingface, at roneneldan/TinyStories-1M/3M/8M/28M/33M/1Layer-21M.
Additional resources:
tinystories_all_data.tar.gz - contains a superset of… See the full description on the dataset page: https://huggingface.co/datasets/HayatoHongo/TinyStories.