ysngkil/whole-books

Whole Books

Five book-length corpora for long-context language-model pretraining, packaged so that one row is one whole book. Together: 202,343 books, about 20.9 B tokens in a 32K-vocabulary Llama-style tokenizer, with the large majority of tokens inside books of 64K tokens or more. Built 2026-09-06 from pinned snapshots of the sources below; nothing was filtered, deduplicated or cleaned beyond what the sources had already done, and the reassembly steps are documented per config.

from datasets import load_dataset
ds = load_dataset("ysngkil/whole-books", "gutenberg", split="train")   # gutenberg | pre1929 | doab | textbooks | pressbooks

Configs

Token counts are estimates: characters times a chars-per-token ratio calibrated on random samples with fla-hub/transformer-1.3B-100B (32,000 tokens, Llama-style). Llama-3-style 128K vocabularies give ~15-20% fewer tokens.

Provenance and licences

gutenberg, pre1929, doab, pressbooks are the _filtered subsets released with Common Pile v0.1 (Kandpal et al. 2025), i.e. already language-filtered, quality-filtered and deduplicated by them. textbooks is the chapter dataset that ProLong (Gao et al. 2024) used for its textbooks domain.

How each config was built

• —gutenberg and pre1929: the Common Pile records converted 1:1 to parquet (one record = one book). The dolma metadata fields are flattened into columns (title, url, language, license, and for pre1929 also author, year, place, htid, hathi_url, ia_url, ...); text is untouched. Note that the PG-19 benchmark (Rae et al. 2019) is ~96% contained in gutenberg by Gutenberg id; exclude PG-19's test ids before evaluating on it.
• —doab: Common Pile stores each DOAB book as several section records (section_idx). Sections of a book are contiguous and ordered in the snapshot; they were concatenated in section_idx order with a blank line between them. 403,992 sections -> 18,948 books; 9,851 books have gaps in the section numbering (sections removed by Common Pile's filter) and are concatenated as-is; 1,050 books share title and author with an earlier book_id and are flagged in dup_of. Columns: book_id, title, author, publisher, isbn, license, url, created, n_sections, section_idx, n_sections_missing, n_chars, dup_of, crawl_order, text.
• —textbooks: TextbookChapters rows are single chapters keyed by a path textbooks/<field>/<shelf>/<Book>/<chapter>; chapters were grouped by book and joined in natural path order. 77,932 chapters -> 1,149 textbooks. Columns: book, shelf, field, n_chapters, n_chars, first_chapter, text. ProLong trained on a stream of these textbooks cut into 512K-token pieces; this config is the uncut superset.
• —pressbooks: Common Pile stores one record per chapter page carrying only the book title; pages were grouped by the URL prefix before /chapter/<slug>/, kept in crawl order, and a # <slug humanised> heading was inserted before each chapter whose slug is not purely numeric. 54,455 pages -> 1,894 books. Crawl order matched the chapter numbering in the slugs in 318 of the 394 books where that could be checked; the live Pressbooks TOC API was not reachable from the build machine, so the order is not verified against it. Columns: book_url, landing_url, title, author, institution, subject, license, n_chapters, n_chars, chapter_slugs, dup_of, crawl_order, text.

Each config directory also holds a _summary.json with the build parameters where a reassembly was done.

Citation of the sources

• —Common Pile v0.1: Kandpal et al., The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text, 2025.
• —TextbookChapters / ProLong: Gao et al., How to Train Long-Context Language Models (Effectively), 2024.
• —PG-19 (for the overlap note): Rae et al., Compressive Transformers for Long-Range Sequence Modelling, 2019.