thesimonharms/public-javanese-dataset

Public-Domain Javanese Manuscript & Text Dataset

A curated collection of public-domain (or openly-licensed) Javanese-language source material — manuscript scans, plain-text transcriptions, digitized printed books, and aksara Jawa (Carakan) primers.

What's in it

Layout

.
├── README.md                 this file (also HF dataset card)
├── LICENSE.md                repo-wide license summary (per-source)
├── LICENSE                   MIT - covers repo code (fetcher/cleaner)
├── CITATION.cff              machine-readable citation
├── CHANGELOG.md              dataset version history
├── sources.yaml              machine-readable manifest of every source
├── .gitattributes            Git LFS rules (PDFs/ZIPs/TIFFs in LFS)
├── scripts/
│   ├── fetch_sources.py      fetch every source in sources.yaml into <dir>/raw/
│   ├── clean_corpus.py       normalize raw OCR text into clean text
│   └── make_readmes.py       regenerate per-source README.md files
└── <source-dir>/
    ├── README.md             provenance + author + license for the source
    ├── raw/                  downloaded files (PDF/TXT/JSON) - in Git LFS
    └── clean/                cleaned/normalized text or .skipped.txt for binaries

Reproducing (smoke self-check)

# 1. (Re)download raw materials into each <dir>/raw/
python scripts/fetch_sources.py
# 2. (Re)clean OCR text into each <dir>/clean/
python scripts/clean_corpus.py
# 3. Regenerate per-source READMEs after editing sources.yaml
python scripts/make_readmes.py

fetch_sources.py is idempotent — re-running skips files that already exist with non-zero size. It needs curl-equivalent reachability to:

• —archive.org (Internet Archive downloads)
• —upload.wikimedia.org (Wikimedia Commons + Wikisource PDFs)
• —jv.wikisource.org (Wikisource API)

Licenses

Per-source licensing is recorded in both sources.yaml and each directory's README.md; see `LICENSE.md` for the summary. The repository code (scripts/*.py, the YAML manifest, and these docs) is MIT — see `LICENSE`.

Aksara Jawa / Carakan script — important limitation

Across this dataset, the Javanese script (aksara Jawa / Carakan) is present as page images (PDF/TIFF/JP2 scans in `<dir>/raw/`), not as Unicode text. There is currently no production-grade public OCR engine for the Carakan script, so Internet Archive's _djvu.txt OCR sidecars devolve to garbage glyphs or Latin transliterations for aksara Jawa passages.

scripts/clean_corpus.py preserves those sidecars as-is (after de-noising page numbers, control characters, runs of spaces, hyphenation joins, and blank-line collapse) so the corpus stays audit-able rather than silently dropping the OCR. For ground-truth transcription of Carakan passages, consult the page scans in each source's raw/ directory.

The Latin-script materials (Old Javanese macapat kakawin introductions, Dutch/French grammar dictionaries, modern Javanese short stories, and the Wikisource index) do have usable machine-readable text in <dir>/clean/.

Vocabulary note

• —Old Javanese / Kawi — language of the kakawin (~9th–15th c.)
• —New Javanese / Modern Javanese — language of the babad, serat
• —Aksara Jawa / Carakan — the Javanese script, present in manuscript scans and primers (blajar-maca-jawa-1996, babad-diponegoro-lor6547).

Known limits

• —Some Internet Archive items do not assert a license string in their metadata; those entries are catalogued under "public domain by age" rationale in sources.yaml. If your use requires definitive correctness, re-confirm at the institution's URL listed in the per-source README before redistributing.
• —A few Wikisource uploads (CC-BY-SA 3.0 / GFDL) require ShareAlike terms to propagate on derivative distributions. The full per-source license is in sources.yaml.
• —Git LFS-tracked large scans are excluded from the GitHub mirror's primary checkout, but are fully present on the Hugging Face dataset repo.

Citation

If you use this dataset, please cite it via CITATION.cff (Zenodo-friendly) or the BibTeX block in this card's YAML frontmatter. Also credit the original institutions per the per-source README.md — this dataset only aggregates and repackages their work.