Ericu950/AncientGreek

Ancient Greek Corpus — re-OCR'd, LLM-repaired

A large, deduplicated Ancient Greek corpus, ~361M words in 2,145,799 records. The bulk of it is public-domain page images that we re-OCR'd ourselves with a vision–language model fine-tuned for polytonic Greek (Angleraud et al., 2026), rather than reusing the existing text layers, which roughly quadrupled the usable yield. The noisier output was then repaired by Qwen3.6-27B.

Provenance

Only the two public-domain page-image collections were re-OCR'd; the born-digital editions are carried through unchanged and credited above. Per-record provenance is in the source and license fields.

Not included: the Database of Byzantine Book Epigrams

DBBE is distributed under CC BY-NC-SA, whose non-commercial clause cannot be honoured by a CC BY-SA compilation. It is therefore excluded from this release (5,476 records, ~0.2M words, 0.1% of the corpus) even though it was part of the corpus used in our own pretraining. To reconstruct that corpus exactly, fetch DBBE yourself under its own terms:

# scripts/fetch_dbbe.py in the code repository
from datasets import load_dataset, concatenate_datasets
ds = load_dataset("Ericu950/AncientGreek", split="repaired")
dbbe = load_dbbe()          # see the script; queries the DBBE API under CC BY-NC-SA
full = concatenate_datasets([ds, dbbe])   # your combined copy is then NC-encumbered

Tiers (splits)

A cleanliness floor (clean ≥ 0.75) excludes the noisiest material, which the LLM could not reliably repair; the per-record clean score lets you tighten it further.

Record schema

{"source": "greek_pd", "id": "...", "license": "Public domain",
 "tier": "repaired", "orig_band": "p080", "clean": 0.83, "text": "..."}

text is NFC-normalised polytonic Greek. clean is the spellchecker score of the OCR before repair; orig_band is its cleanliness band.

How the repaired tier was made

Bands in which 60–95% of words were accepted by an Ancient Greek spellchecker were shown to Qwen3.6-27B with an editor prompt that rejoins words split across lines, reflows one-word-per-line text, restores breathings, accents and iota subscript, fixes only certain OCR letter confusions, expands nomina sacra, and drops any clause it cannot restore with confidence. Genuinely garbled input yields nothing rather than a guess; output word-skeletons were verified against the source, so the editor cannot fabricate text.

The repaired tier is training material, not a critical text. Reusing it as a philological source risks propagating OCR error into scholarship; the pristine tier is untouched if you need text you can quote.

Coverage and representativeness

Ancient Greek (grc), polytonic, archaic through Byzantine, literary prose and verse plus documentary material. The corpus reflects the surviving canon and its transmission biases: it is overwhelmingly the work of elite male authors preserved by later copying, with documentary texts supplying most of what survives from other social strata. Ancient sources treat slavery, sexual violence and ethnic hostility as ordinary; this content is retained, because for philological research it is the object of study.

Reproducibility

Mixed-licence compilation (per-record license field) offered under CC BY-SA 4.0. Pin the revision you load — load_dataset("Ericu950/AncientGreek", revision=...) — so a later update cannot silently change a published result.

Usage

from datasets import load_dataset
ds = load_dataset("Ericu950/AncientGreek")                     # splits: pristine, repaired
clean = load_dataset("Ericu950/AncientGreek", split="pristine")
rep = load_dataset("Ericu950/AncientGreek", split="repaired").filter(lambda r: r["clean"] >= 0.9)