jbduran/bartholomew-dataset-v2

BART Dataset v2

The second version of the BART pretraining corpus, focused on stripping low-quality text — boilerplate and OCR corruption — out of v1.

Lineage — three cumulative filtering stages over the same corpus:

Institutional Books 1.0 → v1 → v2 → v3

How it was made

This phase of filtering targeted boilerplate text and OCR corruption that snuck through Institutional Books' own filters. It removed useless headers, footers and library stamps, alongside gibberish symbols and antiquated characters, through heuristic regex filters.

We also followed Michael Hla's log-prior filter (from Machina Mirabilis / gpt1900), which estimates a text sequence's unconditional log-prior probability by summing token-level log probabilities, then trims improbable or out-of-distribution tokens. Documents were kept inside a moderate GPT-2 token log-prior band of p2.5–p97.5.

Whole books are preserved as rows, and each source shard maps to one output shard of the same basename — so v1's validation shard, shard_00472.parquet, survives as v2's.

The post-1900 physics keyword filter from Hla's original approach was deliberately skipped, because this corpus intentionally keeps texts up to the 1930s.

A note on the result

We expected a major improvement in validation bits-per-byte. Our models actually saw a decline in performance after this phase — because removing the boilerplate removed most of the easy items for a model to replicate. The repetitive headers, footers and stamps were cheap tokens to predict, and stripping them made the remaining corpus genuinely harder. This is worth knowing before you choose v2 over v1: the text is cleaner, but the benchmark number moves the wrong way.

Filtering summary

<details> <summary><b>Log-prior threshold calibration</b></summary>

Thresholds were calibrated on a sample of 3,946 raw documents, of which 3,890 survived structural filtering (56 removed as ocr_artifacts). Estimated prior-based removal from that sample: 5.04%.

Full values in _prior/thresholds.json; per-run detail in cleaning_report.json.

</details>

<details> <summary><b>Character accounting</b></summary>

</details>

Schema and usage

Single string column named text.

from datasets import load_dataset

ds = load_dataset("jbduran/bart-dataset-v2", split="train", streaming=True)
print(next(iter(ds))["text"][:500])

Artifacts: _prior/ (log-prior thresholds and sample stats), cleaning_report.json (full run report).

Citation

Ultimately derived from Institutional Books 1.0:

@misc{institutionalbooks2025,
  title  = {Institutional Books 1.0: A 242B Token Dataset from Harvard Library's
            Collections, Refined for Accuracy and Usability},
  year   = {2025},
  eprint = {2506.08300},
  archivePrefix = {arXiv}
}