exnivo/tinybrain-pretrain-corpus-2b

<p align="center"> <img src="https://huggingface.co/datasets/exnivo/tinybrain-pretrain-corpus-2b/resolve/main/assets/tinybrain-pretrain-banner.png" alt="TinyBrain Pretrain Corpus 2B — Pretraining data for small language models" width="100%" /> </p>

TinyBrain Pretrain Corpus 2B

A mixed-source English pretraining corpus for training small language models.

TinyBrain Pretrain Corpus 2B is a mixed-source dataset built for pretraining small causal language models, especially the TinyBrain-100M Base model.

The dataset combines educational text, factual/wiki-style text, math reasoning data, Python code-summary data, clean web text, and conversation-style data. It is designed to give small models a useful general foundation before supervised fine-tuning.

The uploaded train split contains 3,013,308 rows. A full scan estimates about 7.77B characters, 1.25B words, and roughly 1.81B tokens using a tokenizer-independent estimate. The exact token count may differ depending on the tokenizer used during training.

Most large pretraining corpora are built for much bigger models. TinyBrain Pretrain Corpus 2B is intentionally compact and focused on data that is useful for small models around 100M parameters.

Quick Start

Load the dataset with Hugging Face Datasets:

from datasets import load_dataset

ds = load_dataset("exnivo/tinybrain-pretrain-corpus-2b", split="train")

print(ds)
print(ds[0])

At a Glance

Why Use This Dataset?

TinyBrain Pretrain Corpus 2B is made for people training small language models from scratch.

Use it if you want to:

• —train a small causal language model
• —pretrain a model around 100M parameters
• —experiment with compact pretraining data
• —train a base model before SFT/instruction tuning
• —study how small models learn from educational, factual, math, code, web, and conversation data
• —reproduce or extend the TinyBrain-100M training pipeline
• —build a lightweight local base model

This dataset was used as the base pretraining corpus for `exnivo/tinybrain-100m-base`.

Dataset Summary

TinyBrain Pretrain Corpus 2B is a mixed-source English pretraining dataset.

The dataset was built to balance useful general knowledge with small-model-friendly sources. It includes educational web text, factual reference text, math reasoning data, code-related data, clean web text, and conversation-style examples.

The goal is not to create the largest possible web scrape. The goal is to create a compact, useful, and varied pretraining corpus that gives a small model enough language, factual, math, coding, and dialogue exposure before instruction tuning.

Each row contains text plus basic metadata:

{
  "text": "...",
  "category": "...",
  "source": "..."
}

Real Dataset Stats

Source Breakdown

Category Breakdown

Source × Category Breakdown

Text Length and Token Estimate

Token counts are approximate and estimated without a tokenizer. For exact token counts, tokenize the dataset with the same tokenizer used during training.

Dataset Structure

Each row contains one text document or text chunk.

Main fields:

Example row:

{
  "text": "Photosynthesis is the process by which plants use sunlight, water, and carbon dioxide to make food...",
  "category": "educational",
  "source": "FineWeb-Edu sample-10BT"
}

Data Composition

TinyBrain Pretrain Corpus 2B is built from several broad data types.

Format

The dataset is stored as text data with metadata.

A typical row looks like:

{"text": "...", "category": "...", "source": "..."}

For causal language model pretraining, the text field is usually tokenized and packed into fixed-length token blocks.

Convert to Plain Text

If you want a simple text-only dataset for tokenization:

from datasets import load_dataset

ds = load_dataset("exnivo/tinybrain-pretrain-corpus-2b", split="train")

def keep_text(example):
    return {"text": example["text"]}

text_ds = ds.map(
    keep_text,
    remove_columns=[c for c in ds.column_names if c != "text"]
)

print(text_ds[0]["text"])

Inspect the Dataset

You can inspect the sources and categories with:

from datasets import load_dataset
from collections import Counter

ds = load_dataset("exnivo/tinybrain-pretrain-corpus-2b", split="train")

print(ds)
print(ds.column_names)
print(ds[0])

print("\nSource counts:")
for name, count in Counter(ds["source"]).most_common():
    print(name, count)

print("\nCategory counts:")
for name, count in Counter(ds["category"]).most_common():
    print(name, count)

Check text lengths:

lengths = [len(x["text"]) for x in ds]

print("Min chars:", min(lengths))
print("Max chars:", max(lengths))
print("Average chars:", sum(lengths) / len(lengths))

Preview samples:

import random

for i in random.sample(range(len(ds)), 5):
    row = ds[i]
    print("source:", row.get("source"))
    print("category:", row.get("category"))
    print(row["text"][:1000])
    print("-" * 80)

Example Pretraining Use

TinyBrain Pretrain Corpus 2B is intended for standard causal language modeling.

A typical pretraining flow is:

1. 1.Load the dataset.
2. 2.Read the text field.
3. 3.Train or load a tokenizer.
4. 4.Tokenize all text.
5. 5.Pack tokens into fixed-length blocks.
6. 6.Train a causal language model with next-token prediction.
7. 7.Evaluate on held-out validation data.
8. 8.Optionally fine-tune the base model with an SFT dataset.

Example loading setup:

from datasets import load_dataset
from transformers import AutoTokenizer

dataset_id = "exnivo/tinybrain-pretrain-corpus-2b"

ds = load_dataset(dataset_id, split="train")

tokenizer = AutoTokenizer.from_pretrained("exnivo/tinybrain-100m-base")

def tokenize(example):
    return tokenizer(example["text"])

tokenized = ds.map(
    tokenize,
    remove_columns=ds.column_names,
    num_proc=4
)

print(tokenized[0])

Relationship to TinyBrain Models

This dataset is the pretraining corpus for the TinyBrain model pipeline.

The intended pipeline is:

TinyBrain Pretrain Corpus 2B
        ↓
TinyBrain-100M Base
        ↓
TinyBrain Instruct 200K
        ↓
TinyBrain-100M Instruct

Real Samples

These are short examples from the uploaded dataset.

Educational

Image of the Month - April 2017 The Image of the Month for April 2017 is one of our favourite galaxies, Messier 88 (M88), or NGC4501. This is a spiral galaxy which lies over 50 million light years from Earth, sitting within a cluster of galaxies known as the Virgo Cluster.

Factual

Question: Explain Facebook F8 in simple words. Answer: Facebook F8 is a mostly-annual conference held by Meta Platforms (formerly Facebook) since 2007, intended for developers and entrepreneurs who build products and services around the website.

Math Reasoning

Problem: What is the smallest positive integer that is both a multiple of 11 and a multiple of 5? Solution: To find the smallest positive integer that is both a multiple of 11 and a multiple of 5, we need to find the least common multiple (LCM) of 11 and 5. The first number that appears in both lists is 55.

Code

Language: python Description: Utility used to make sure AST parser does not choke on unrecognized magics. Code: def comment_out_magics(source): """ Utility used to make sure AST parser does not choke on unrecognized magics. """ filtered = [] for line in source.splitlines(): if line.strip().startswith('%'): filtered.append('# ' + line) else: filtered.append(line) return '\n'.join(filtered)

Conversation

User: Sammy has 2 more bottle caps than Janine. Janine has 3 times as many bottle caps as Billie. If Billie has 2 bottle caps, how many does Sammy have? Assistant: If Billie has 2 bottle caps, Janine has 3 times as many, so Janine has 6 bottle caps. Sammy has 2 more than Janine, so Sammy has 8 bottle caps.

Data Quality Snapshot

A full scan of the uploaded train split found:

The dataset has no empty text rows, no missing source/category rows, and no exact duplicate text rows in the scan. Users who want stricter training may still want to filter bad-pattern rows, HTML-like rows, very long rows, and near-duplicates.

Bad Pattern Matches

Data Quality Notes

The dataset is a curated mixed-source corpus, but users should still inspect and filter the data before serious training.

Recommended checks before training:

• —source distribution
• —category distribution
• —text length distribution
• —exact duplicates
• —near-duplicates
• —unwanted boilerplate
• —HTML/script leftovers
• —non-English rows
• —license compatibility
• —source-specific quality issues

Example quality check:

from datasets import load_dataset
from collections import Counter

ds = load_dataset("exnivo/tinybrain-pretrain-corpus-2b", split="train")

empty = 0
short = 0
bad = 0

for row in ds:
    text = str(row.get("text", "")).strip()
    if not text:
        empty += 1
    if len(text) < 50:
        short += 1
    if "\x00" in text:
        bad += 1

print("empty rows:", empty)
print("short rows:", short)
print("rows with null chars:", bad)
print("sources:", Counter(ds["source"]).most_common())

Intended Use

TinyBrain Pretrain Corpus 2B is intended for:

• —small language model pretraining
• —causal language modeling experiments
• —educational language modeling
• —math/reasoning pretraining
• —training compact base models
• —studying small-model data mixtures
• —reproducing TinyBrain-100M Base-style experiments

This dataset is not meant to be a finished assistant dataset. It is a base pretraining corpus. For chat/instruction behavior, use an SFT dataset after pretraining.

Not Intended For

This dataset is not intended to be used directly for:

• —instruction/chat fine-tuning by itself
• —high-stakes factual systems
• —medical, legal, financial, or safety-critical applications
• —live/current factual information
• —perfectly deduplicated web-scale training
• —benchmark-grade math training alone
• —production systems without further filtering and evaluation

A model pretrained on this dataset may still need instruction tuning, safety tuning, evaluation, and additional cleanup depending on the target use case.

Strengths

TinyBrain Pretrain Corpus 2B is useful because it is:

• —compact compared to large web-scale corpora
• —focused on small language models
• —mostly English
• —mixed across factual, educational, math, code, conversation, and clean web data
• —designed for TinyBrain-100M-style pretraining
• —large enough to train a small base model
• —easier to inspect and reason about than massive pretraining datasets
• —linked to a complete pipeline with base and instruct models

Limitations

This dataset has limitations.

The dataset may contain:

• —factual mistakes
• —outdated information
• —duplicate ideas or near-duplicates
• —noisy web text
• —HTML or boilerplate leftovers
• —privacy/cookie policy fragments
• —uneven source balance
• —artifacts from upstream datasets
• —incomplete metadata
• —mixed licensing constraints
• —content that may not be ideal for all use cases

The token count is approximate and based on a tokenizer-independent estimate, not necessarily the exact tokenizer count used during training.

Because this is a base pretraining dataset, it does not by itself teach strong assistant behavior, refusal behavior, or instruction-following. Those behaviors are expected to come later through supervised fine-tuning.

Suggested Evaluation

Models pretrained on this dataset should be evaluated before and after SFT.

Useful checks include:

• —validation loss / perplexity
• —short factual completions
• —arithmetic completions
• —simple reasoning prompts
• —repetition tests
• —memorization checks
• —code completion sanity checks
• —hallucination checks
• —instruction-following after SFT
• —comparison against the same model before/after instruction tuning

Example base-model prompts:

Paris is the capital city of

The Netherlands is a country in

A cat is an animal that

One plus one equals

Photosynthesis is the process by which plants

For chat behavior, use an instruction-tuned model such as `exnivo/tinybrain-100m-instruct`.

Recommended Dataset Mixing

If you build a new version of this corpus, possible improvements include:

For small models, data quality matters more than just increasing dataset size.

Version Notes

This is an early TinyBrain pretraining corpus release.

Future versions may include:

• —clearer train/validation/test splits
• —stronger deduplication
• —exact tokenizer-based token counts
• —improved source metadata
• —quality scores
• —cleaner license metadata
• —more balanced source mixing
• —better code subset
• —more curated math subset
• —stricter boilerplate filtering
• —smaller sample/demo version
• —direct tokenized shards for faster training

Citation

If you use this dataset, you can cite it as:

@misc{tinybrain_pretrain_corpus_2b,
  title = {TinyBrain Pretrain Corpus 2B},
  author = {exnivo},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/exnivo/tinybrain-pretrain-corpus-2b}}
}

Related Repositories

• —Pretraining corpus: `exnivo/tinybrain-pretrain-corpus-2b`
• —Base model: `exnivo/tinybrain-100m-base`
• —SFT dataset: `exnivo/tinybrain-instruct-sft-200k`
• —Instruct model: `exnivo/tinybrain-100m-instruct`

License

The dataset license is currently listed as other.

This is intentional for now. TinyBrain Pretrain Corpus 2B is a mixed-source dataset built from multiple upstream public datasets with different licenses and terms.

Users are responsible for checking and following the licenses of the original source datasets before using this corpus, especially for commercial use.

A future release may move to clearer source-level license metadata if upstream source compatibility is fully verified.

Disclaimer

TinyBrain Pretrain Corpus 2B is an experimental mixed-source pretraining dataset. It may contain mistakes, noise, duplicates, outdated information, boilerplate, or low-quality samples from upstream datasets.

Models trained on this dataset may produce incorrect, biased, unsafe, or misleading outputs. Always evaluate models carefully before using them in real applications.