mzzhang2014/glyph_machina_medieval_lines

glyphmachinamedieval_lines

Noisy HTR pretraining set: text-line crops from pre-Elizabeth-I English legal manuscripts (AALT scans), with machine-generated transcriptions (confidence prefixes stripped, confidence-filtered upstream). Line images are dewarped, background-subtracted, inverted, 64 px tall.

Format: page-grouped WebDataset

data/*.tar are WebDataset shards (~1 GB each). One sample = one page. For a page whose key is e.g. H4-E159no176-bE159no176dorses-IMG_0698:

H4-E159no176-bE159no176dorses-IMG_0698.00.png   line 0 (reading order, y-sorted)
H4-E159no176-bE159no176dorses-IMG_0698.01.png   line 1
...
H4-E159no176-bE159no176dorses-IMG_0698.json     {"texts": ["...", "...", ...]}

texts[i] is the transcription of {key}.{i:02d}.png; up to 80 lines per page. Shards named b####.tar mirror the extraction batches; loose-####.tar hold the original run. Pages never straddle shards. The dataset is being extended — new b####.tar shards appear as extraction proceeds.

Streaming on a GPU cluster (no local storage)

Use the webdataset library — it streams tars over HTTP and yields one dict per page. (Prefer it over datasets.load_dataset("webdataset", ...) here: pages have varying line counts, so samples have heterogeneous columns, which the HF builder dislikes but webdataset handles natively.)

import io, json
import webdataset as wds
from huggingface_hub import get_token, HfApi

repo = "mzzhang2014/glyph_machina_medieval_lines"
names = [f for f in HfApi().list_repo_files(repo, repo_type="dataset")
         if f.endswith(".tar")]
urls = [f"pipe:curl -sfL -H 'Authorization: Bearer {get_token()}' "
        f"https://huggingface.co/datasets/{repo}/resolve/main/{n}" for n in names]

def to_page(sample):
    keys = sorted(k for k in sample if k.endswith(".png"))
    return {"__key__": sample["__key__"],
            "images": [sample[k] for k in keys],          # raw PNG bytes
            "texts": json.loads(sample["json"])["texts"]}

ds = (wds.WebDataset(urls, shardshuffle=True, nodesplitter=wds.split_by_node)
      .map(to_page))

Wrap in wds.WebLoader for multi-worker loading; shards are the unit of shuffling/distribution across nodes. Decode PNGs with PIL.Image.open(io.BytesIO(b)) (mode L, height 64, white-on-black).