LisaMegaWatts/bookcorpus-gutenberg-classics

BookCorpus + Gutenberg Classics Training Corpus

Large-scale training corpus combining BookCorpus fiction, Project Gutenberg 19th-century literature (PG-19), and curated classical philosophy texts. Cleaned, deduplicated, and organized into curriculum phases for character-level language model training.

Dataset Description

This corpus is the primary training dataset for the Julia SLM project, combining three major text sources into a unified, cleaned training set with curriculum-phase annotations for structured learning.

Source Composition

Cleaning Applied

All text has been processed through a multi-stage cleaning pipeline:

• —Character filtering: Lowercased to ASCII set a-z .,;:?!'"()-
• —Source-specific cleaning:
• —BookCorpus: Moses-style detokenization (recombined subword artifacts)
• —PG-19: Gutenberg boilerplate header/footer removal
• —Philosophy: LaTeX artifact removal, footnote/reference stripping
• —Deduplication: Exact dedup removed 803,221 duplicates (5.1%)
• —Whitespace normalization: Multi-space collapse, empty line removal

Curriculum Phases

The corpus is organized into three curriculum phases based on the classical trivium/quadrivium education model, suitable for DoReMi-style weighted phase sampling:

Phase-specific training files are available in the curriculum/ directory.

Dataset Statistics

• —90/10 train/validation split (shuffled)
• —Weighted phase sampling applied per config: trivium 40%, quadrivium 35%, philosophy 25%

Philosophy Sources

The corpus includes texts from 50+ classical authors spanning Greek, Roman, Medieval, Enlightenment, and Modern philosophy:

Greek: Aristotle (Metaphysics, Nicomachean Ethics, Politics, Physics, Rhetoric, Poetics, Categories, Prior/Posterior Analytics, Topics, On the Soul, On the Heavens, On Interpretation, Generation and Corruption), Plato (Republic, Laws, Timaeus, Phaedo, Phaedrus, Symposium, Meno, Theaetetus, Protagoras), Herodotus, Thucydides, Xenophon, Aeschylus, Sophocles, Homer, Euripides

Roman: Marcus Aurelius, Seneca, Epictetus, Cicero, Lucretius, Plutarch, Tacitus, Virgil

Medieval/Renaissance: Boethius, Machiavelli, Thomas More

Enlightenment: Descartes, Spinoza, Leibniz, Locke, Berkeley, Hume, Kant, Rousseau, Montesquieu

Modern: Schopenhauer, Mill, Thoreau, William James

Eastern: Bhagavad Gita, Sun Tzu, Confucius, Lao Tzu

Additional Files

The curriculum/ directory contains phase-specific training files:

• —train_trivium.txt - Grammar, rhetoric, and logic texts (2.0 GB)
• —train_quadrivium.txt - Mathematical and natural philosophy texts (2.1 MB)
• —train_philosophy.txt - Pure philosophy texts (13 MB)

Usage

from datasets import load_dataset

ds = load_dataset("LisaMegaWatts/bookcorpus-gutenberg-classics")

# Training data
for example in ds["train"]:
    text = example["text"]

# Download phase-specific files for curriculum training
from huggingface_hub import hf_hub_download

trivium = hf_hub_download(
    "LisaMegaWatts/bookcorpus-gutenberg-classics",
    "curriculum/train_trivium.txt",
    repo_type="dataset",
)

Related Datasets

• —LisaMegaWatts/philosophy-corpus - Isolated philosophy provenance dataset (for model training lineage)
• —LisaMegaWatts/wikitext-103-quality-scored - Quality-scored WikiText-103 (Wikipedia Featured articles)
• —LisaMegaWatts/classical-humanities-corpus - Extended classical humanities collection

License

Apache 2.0