jbduran/bartholomew-dataset-v3

BART Dataset v3

The final version of the BART pretraining corpus, focused on removing anything that betrays a post-1930 origin. This is our best vintage dataset yet.

Lineage — three cumulative filtering stages over the same corpus:

Institutional Books 1.0 → v1 → v2 → v3

How it was made

For our final phase, we decided to focus on the time period. Although we filtered by date at v1, leaks and bad labeling inevitably brought modern documents in to contaminate the vintage corpus — publication metadata says nothing about a 1990s reprint's front matter, or an OCR pass that swept in a modern colophon.

So we filtered through document boilerplate once more, focusing on footers missed by earlier passes. But the biggest part of this phase was the regex anachronism filter: a tiered banned words list that flags documents and removes them on too many hits.

Rather than trimming, we removed entire documents when the conditions were met. A single Tier 1 hit was enough to discard a document, while Tier 2 words required two occurrences. Tier 3 words never triggered removal on their own — they only counted when accompanied by at least one Tier 2 hit.

We used this same strategy later in midtraining, to keep all of our data as vintage as possible.

The tiered anachronism filter

A document is dropped only on strong evidence of post-1930 content. The tiering exists to avoid polysemy false positives — "compiler of this volume", a bee "drone", birdsong "twitter", the Black Hole of Calcutta.

Drop rule: 1× tier1 OR (≥2 distinct tier2/tier3 with ≥1 tier2)

The list was built from a 500-term seed, then curated: a 165-entry allow-list removed 55 seed terms, 18 were added by hand, leaving 463 final terms plus 10 format-tell patterns.

Stage 0.5 — footer and boilerplate stripping

Runs before the anachronism filter. A line-level pass removes reprint and OCR footer lines — URLs, "printed in the United States of America", "all rights reserved", photocopy and print-on-demand colophons, ISBN lines, bare page numbers, library stamps — writing the stripped corpus to stripped/. Whole books are kept; only footer lines are removed. Documents that would lose more than 30% of their lines are kept unstripped and flagged.

<details> <summary><b>Footer patterns by frequency</b></summary>

</details>

<details> <summary><b>Top firing anachronism terms</b></summary>

Terms that most often flagged a document. Note how many are polysemous — which is exactly why the tiering and allow-list exist.

</details>

Schema and usage

Single string column named text.

from datasets import load_dataset

ds = load_dataset("jbduran/bart-dataset-v3", split="train", streaming=True)
print(next(iter(ds))["text"][:500])

Artifacts:

• —_banned/ — the tiered lists, allow-list, format tells, and build audit
• —stripped/ — the footer-stripped corpus
• —strip_stats/, strip_samples/ — footer-strip audit
• —stats/ — per-shard filter counts
• —hits/ — per-shard hit log (which terms fired on which documents)
• —scripts/ — the pipeline
• —cleaning_report_1930s.json — full run report

Citation

Ultimately derived from Institutional Books 1.0:

@misc{institutionalbooks2025,
  title  = {Institutional Books 1.0: A 242B Token Dataset from Harvard Library's
            Collections, Refined for Accuracy and Usability},
  year   = {2025},
  eprint = {2506.08300},
  archivePrefix = {arXiv}
}