jbduran/bartholomew-dataset-v1

BART Dataset v1

The first version of the BART pretraining corpus: pre-1930 English books drawn from Institutional Books 1.0 and filtered hard on OCR quality, language, date, and tokenizability.

Lineage — three cumulative filtering stages over the same corpus:

Institutional Books 1.0 → v1 → v2 → v3

How it was made

We kept searching for a vintage pretraining corpus and came across Institutional Books 1.0, the product of the Harvard Library's digitization and post-processing work — 242B tokens across almost a million documents.

Working from Institutional Books, we applied an aggressive OCR score threshold of 0.90, drawing on both OCRoscope and the OCR score from Google Books metadata, and required that the two metrics agree within 0.10 on any given document. We then filtered by date, rejecting any undated book outright; filtered for English, which produced by far the most rejects; and dropped text with a tokenizability score below 95 — a measure of how efficiently a tokenizer encodes the text.

This trimmed 242B tokens down to roughly 27B. We then deduplicated by book barcode and shuffled the dataset. The result is BART Dataset v1.

Filters

Deduplication — via likely_duplicates_barcodes_gen barcode claiming; the first-seen representative is kept.

Diversity — a single shuffle buffer with uniform random eviction. Shards 00000-00449 used a 20 GB buffer; the final 23 shards, 00450-00472, used an 8 GB buffer to reduce Colab memory pressure. Books were randomly evicted once the active buffer exceeded its limit, reducing source-order clustering while preserving the corpus's natural topic distribution.

Validation split — deterministic by barcode hash: a book is val iff crc32(barcode) % 370 == 0 (~1,000 books). Resume-stable, and written as the last lexicographic shard, shard_00472.parquet.

Topic distribution

Per-shard topic distributions live in manifest.json under shards[].topic_distribution.

<details> <summary><b>Rejection counts and the language-filtering estimate</b></summary>

Raw rejection tallies, which are not exact — interrupted resumes counted some rejected source rows more than once:

Estimated language filtering. Because exact totals weren't preserved, a clean 50,000-row dry run using the final filters was used instead: it rejected 18,851 non-English books, 8,772 books below the 90% English threshold, and 9 with missing language metadata. Extrapolated across 983,004 source rows, approximately 543,200 books (55.3%) were removed by language filtering. These are estimates, not exact counts.

Pass by date source (date1_src vs date2_src fallback recovery): date1_src 202,458, date2_src 12.

</details>

Schema and usage

Single string column named text. ZSTD-3 compression, row-group size 64. Per-document audit metadata lives separately in audit_metadata.jsonl; the training shards stay text-only.

from datasets import load_dataset

ds = load_dataset("jbduran/bart-dataset-v1", split="train", streaming=True)
print(next(iter(ds))["text"][:500])

Use with nanochat

Drop-in compatible with nanochat/dataset.py. Rename the directory to base_data_climbmix/ to match DATA_DIR in nanochat/dataset.py:27, or edit that constant to point at this directory. Then re-train the tokenizer:

python -m scripts.tok_train && python -m scripts.tok_eval
python -m scripts.base_train --depth=12 --window-pattern=L

Citation

This dataset is derived from Institutional Books 1.0:

@misc{institutionalbooks2025,
  title  = {Institutional Books 1.0: A 242B Token Dataset from Harvard Library's
            Collections, Refined for Accuracy and Usability},
  year   = {2025},
  eprint = {2506.08300},
  archivePrefix = {arXiv}
}