CoolFace
Datasetpublic

mhla/pre1900-corpus

Pre-1900 Corpus The training corpus for GPT-1900 — a cleaned collection of pre-1900 English-language texts with full metadata. Every document in this corpus was published before the year 1900. Schema Column Type Description text string Full document text year int64 Publication year title string Book title or newspaper name source string Source dataset identifier ocr_score float64 OCR confidence score (-1.0 if unavailable) legibility float64… See the full description on the dataset page: https://huggingface.co/datasets/mhla/pre1900-corpus.

sourceHugging Facemitupdated 6mo agoView on Hugging Face
4likes336downloads
Dataset Card

Pre-1900 Corpus

The training corpus for GPT-1900 — a cleaned collection of pre-1900 English-language texts with full metadata. Every document in this corpus was published before the year 1900.

Schema

ColumnTypeDescription
textstringFull document text
yearint64Publication year
titlestringBook title or newspaper name
sourcestringSource dataset identifier
ocr_scorefloat64OCR confidence score (-1.0 if unavailable)
legibilityfloat64Legibility score (-1.0 if unavailable)

Sources

  • —Institutional books — HathiTrust, Internet Archive, and other digitized book collections
  • —British Library books — TheBritishLibrary/blbooks
  • —Historical newspapers — dell-research-harvard/AmericanStories

Filtering Pipeline

  1. 1.OCR cleanup — removal of OCR artifacts, boilerplate, and unicode normalization
  2. 2.Quality filtering — token frequency prior-based filtering as a cheap proxy for perplexity
  3. 3.Anachronism detection — three-tier post-1900 physics filter to remove mislabeled modern texts:
  4. 4.Always reject: unambiguous post-1900 terms (photon, spacetime, transistor, etc.)
  5. 5.Date reject: documents with 5+ explicit post-1900 year references
  6. 6.Context reject: 3+ co-occurring ambiguous terms (quantum, nuclear, radiation, etc.)

Usage

python
from datasets import load_dataset
ds = load_dataset("mhla/pre1900-corpus")

Related