Gugu8/Pretrain

pretrain-60GB

A 60GB, knowledge-dense pretraining corpus built as a clean, superior alternative to TinyStories.

TinyStories teaches grammar. This teaches knowledge, reasoning, and code.

File: pretrain.csv - 60GB, ~15M rows, single column text

Why vs TinyStories?

TinyStories is fiction for kids. This is textbook / encyclopedia / reasoning / code. Every row is 3500-5200 chars packed with 2-3 fused concepts, not a story.

• —Physics, Chem, Bio, Math, CS, History, Geo, Econ
• —Step-by-step math with verification
• —Algorithms with invariants + edge cases
• —Logic puzzles with backtracking traces

All synthetic, original, cleaned. One line per example for fast streaming. No scraped data, no copyrighted text.

Usage

Streaming is recommended for 60GB:

from datasets import load_dataset from transformers import AutoTokenizer

ds = loaddataset("YOURUSERNAME/pretrain-60GB", streaming=True, split="train")

for row in ds: print(row['text'][:500]) break

tok = AutoTokenizer.frompretrained("gpt2") def tokenize(ex): return tok(ex["text"], truncation=True, maxlength=1024)

ds_tokenized = ds.map(tokenize)

With PyTorch:

import pandas as pd for chunk in pd.read_csv("pretrain.csv", chunksize=100000): # your training loop pass

Stats

• —Size: 64.4 GB on disk (60 GiB target)
• —Rows: ~15-18M
• —Avg length: ~4000 chars / 600-900 tokens
• —Format: CSV, header text, quoted, UTF-8
• —License: Apache 2.0

Limitations

Synthetic data - may contain simplified explanations. Intended as a base pretrain, fine-tune on curated data after.

License

This dataset uses the Open Data Attribution Training Disclosure License (ODATL‑1.0). You may learn more looking at the LICENSE file.