Datasetpublic

quasar7/pre1900-catalogue

Pre-1900 English & German Catalogue Universal catalogue of pre-1900 (or undated) English and German works: 2,344,644 works, 2,047,100 with full text (869.4GB). Browse & read: https://quasar7-pre1900-catalogue-site.static.hf.space (serverless — searches and reads run in your browser against this repo) Query here: use the SQL Console on this page (the catalogue config is the works table) Layout path contents site/catalog_flat.parquet the catalogue: one… See the full description on the dataset page: https://huggingface.co/datasets/quasar7/pre1900-catalogue.

sourceHugging Facecc0-1.0updated 1mo agoView on Hugging Face

0likes24downloads

Dataset Card

Pre-1900 English & German Catalogue

Universal catalogue of pre-1900 (or undated) English and German works: 2,344,644 works, 2,047,100 with full text (869.4GB).

—Browse & read: https://quasar7-pre1900-catalogue-site.static.hf.space (serverless — searches and reads run in your browser against this repo)
—Query here: use the SQL Console on this page (the catalogue config is the works table)

Layout

path	contents
`site/catalog_flat.parquet`	the catalogue: one row per work (title, author, year, language, quality tier, source, status, chars)
`site/text_locator_all.parquet`	work_id → which text file holds its full text
`text/…`	full texts, one row per book (`work_qid, corpus, ref, text`); `text/fast/…` are small re-cut files for fast single-book reads
`corpus_db8/part_00..20.bin`	the master DuckDB database in 8GiB pieces — `cat part_*.bin > corpus.duckdb` (175,734,009,856 bytes, SHA256-verified)
`aux/`	author database, HathiTrust catalogue data, release backups
`site/catalog_app.py`	the original server app (runs anywhere with the above files)

Read one book from anywhere

sql

-- 1. find the work and its text file
SELECT l.shard FROM 'hf://datasets/quasar7/pre1900-catalogue/site/text_locator_all.parquet' l
WHERE l.work_id = 'ht:uc1.b4534619';
-- 2. fetch the text (replace /data/text/ with .../resolve/main/text/)
SELECT text FROM '<that file URL>' WHERE work_qid = 'ht:uc1.b4534619';

Texts carry page structure as whole-line markers ([[p N · folio]], [[vol …]]); strip them with the regex ^\[\[(?:p|vol)\b[^\]\n]*\]\]$\n?.

Companion repo (raw preservation: HathiTrust page-zips, METS, manifests): `quasar7/pre1900-academics-corpus`