TRACCERR/Sumtables-Cuneiform-Full-Fable5-Remaster

Sumtablets-Cuneiform-Full-Fable5-Remaster — Cuneiform Vision-Language Training Dataset

A rebuilt, leakage-proof, multi-task training dataset for teaching vision-language models (target: Qwen3-VL-8B-Instruct LoRA) to visually read, transliterate, and translate Sumerian cuneiform tablets from photographs. The mission: produce useful first-pass readings for the ~90% of excavated tablets that have never been published or translated.

Current release: v1.0.0 — 455,506 records (402,004 train / 24,507 validation / 24,446 test / 4,549 external grounding), built on 52,602 tablets with at least one tier-A/B image and 87,764 aligned surface-text pairs. Everything below is reproducible from the public pipeline in this repository.

AI disclosure: this dataset restructuring was designed, implemented, and executed by Claude Fable 5 (Anthropic) operating the Sumtablets_v2 pipeline under human direction, with human review at the quality gates (segmentation review, pollution labeling, Gold-set approval). Every automated decision is recorded in per-phase manifests and reports so third parties can validate or contest it.

1. What this is (and what it replaces)

The starting point was `TRACCERR/Sumtablets_Merged` (~200k rows, 53.7 GB, rev 67e3d17), itself a merge of `colesimmons/SumTablets` (glyph–transliteration pairs; paper) and colesimmons/SumTablets_Photos. The original improvement plan was a single re-split tool to fix suspected train/test tablet leakage.

That plan was reviewed and rebuilt into a 7-phase pipeline (Sumtables-Cuneiform-Full-Fable5-Remaster_Dataset_Plan.md) because split hygiene alone left the dataset's real limitations untouched:

2. Pipeline phases — planned vs. delivered

Phase 0 — Audit & leakage-proof re-split (clean_sumtablets.py) ✅

Deterministic tablet-grouped 90/5/5 re-split (seed 42) of all 199,964 rows / 89,485 canonical P###### tablets, stratified by period × genre × modality with minimum eval representation per stratum.

Results (full corpus, all hard assertions PASS):

• —train 178,116 / validation 9,864 / test 9,865 / quarantine 2,119 rows; tablets 80,567 / 4,459 / 4,459 = 90.03 / 4.98 / 4.98%.
• —53,926 exact-duplicate image groups (4 spanning different tablet IDs — flagged for review); 922,737 pHash near-duplicate pairs → 12,481 tablets merged into shared "leak groups" so perceptual twins can never straddle splits. dHash confirmation rejects spurious pHash links (hand-drawn lineart otherwise over-merges).
• —Two giant Ur III similarity components (8,085 and 4,037 tablets) exist; the allocator forces any group larger than the eval quota into train (regression-tested) — largest group in val/test is 2 tablets.
• —Formula leakage measured, not hidden: 156 near-verbatim text groups span tablets; 12 validation and 25 test tablets have a train twin. Tagged in formula_leakage.csv; the eval harness reports metrics with and without them.
• —Quarantine = 2,081 ORACC Q###### composite-text IDs + 38 X###### — real identifiers of composite editions, not physical tablets; excluded from splits by design (re-admission with leak screening is a documented future option).
• —Audit columns on every row: canonical_tablet_id, original_tablet_id, source_split, assigned_split, modality, duplicate_group, formula_dup_group, validation_flags, provenance.

Phase 1 — Image elevation (phase1_images.py) ✅ complete

• —CDLI full-resolution re-fetch, complete census: all 178,970 tablet×kind URL pairs resolved — 71,852 images downloaded (37,810 photos + 34,042 lineart, 116 GB), 106,999 definitive "no image hosted" 404s recorded per tablet, 119 corrupt-at-source files logged. Resumable manifest survived a disk-full crash and an external process kill with zero loss (supervised auto-restart loop; manifest surgery for the ~380 transient errors — note --retry-errors re-attempts 404s too, so targeted retries edit the manifest instead). Finding: re-fetching does not upscale existing images (where CDLI hosts a file the dataset already embedded it at identical resolution); its value is coverage — thousands of images for tablets that had none of that kind, converting text-only tablets into vision samples.
• —Composite segmentation by projection profiling → per-surface panel bounding boxes (manifests only; crops materialize at packaging). Adjustment after human review (199/200 correct, gate G1 passed): stacked CDLI layouts needed a relaxed-gap re-split pass, and the dominant panel may never be labeled an "edge" (regression test P201395).
• —Surface–text alignment: transliterations split at <SURFACE> markers and paired with panels when counts match → 87,764 aligned surface pairs full-corpus (37,410 count-matched + 6,631 trivial images; 34.9% of the 126,282 analyzed images fully align, the rest keep whole-image supervision).
• —Readability tiers A/B/C from blur, contrast, and estimated sign height; tier C never becomes an OCR target. Full-corpus result: 52,602 tablets hold at least one tier-A/B image (47,427 train / 2,598 val / 2,577 test); tier A dominates the fetched images (45,286 of 64,157 train fetches).
• —Document-scan detector — entirely human-review-driven addition: users of the G1 review page found publication text pages mislabeled as lineart. Detector calibrated on 13 human-labeled images (4 pages / 9 genuine hand copies); naive heuristics false-positived at 25–45% (curved hand-drawn outlines defeat vertical-run tests; stacked views shrink per-view runs). Final rule (band geometry + ink density + dilated vertical-run) flags 1,164 images full-corpus (~2.1% of lineart), catching all human-found pages, clearing all 9 labeled copies; spot-check precision 3/4 (over-flagging is accepted: a flag only removes an image from vision training, never deletes data).

Phase 2 — Task architecture (phase2_tasks.py + mixer_config.yaml) ✅

Task-tagged records with 25 deterministic prompt paraphrases per task, canonical output formats validated on every record, per-tablet caps, and a train-only mixture spec.

Adjustment (bug found in v1 mixer): scarce tasks throttled the whole mixture (T8's 5.5k records nearly cut train from ~105k to 12.7k). Fixed semantics: the most-available task anchors totals, scarce tasks underfill with reported deviations, and exact mixture shares are enforced at training time via sampling weights using the shipped mixer_config.yaml.

v1.0.0 totals: 402,004 train records (259,457 vision / 142,547 text) plus 4,549 external T5 grounding records; 24,507 validation and 24,446 test records (~15.4k vision each). Per-task counts live in the release's build_report.json and MANIFEST.parquet. A curated 4,850-record starter pack (starter_train.parquet, vision-first mixture, tier-A/B only for OCR, refusal-capped abstention) ships for quick fine-tune runs.

Phase 3 — Translation layer (phase3_translations.py) ✅

• —CDLI ATF harvest: 5,357 tablets with line-aligned #tr.en translations (2,094 train / 124 val / 121 test in-corpus; 3,018 external tablets admitted to train only after screening their text against every eval tablet's normalized transliteration).
• —Templated Ur III renderer: precision-first CFG rules (only tablets with ≥80% of lines fully parsed) → 409 tablets, source=templated, train-only, 7% of T8 (cap: 40%). Adjustment: rules had to be rewritten in the corpus's Unicode orthography (š, subscript numerals) — the CDLI ASCII convention (sz, plain digits) matches zero corpus lines.
• —Gold-997: stratified test-pool benchmark selection (856 with images, 43 with harvested translations), human-approved selection, per-item verification_status tracking (verification pending).
• —Dead end documented: ORACC etcsri JSON exposes word glosses only — sentence translations are HTML-only (future scrape).

Phase 4 — External sign grounding (phase4_grounding.py) ✅

eBL cuneiform-OCR coco-recognition set (Zenodo 10693601): 654 tablet photographs, 46k+ sign boxes, 120 sign classes → 4,549 T5 records (dense detection, locate-sign, read-region; 0–1000 normalized boxes; valid-JSON targets). This file is 100% vision by construction — grounding has no text-only variant — and stays a separate parquet so the unlicensed images can be excluded from redistribution by omitting one file. Leak-guarded against eval P-numbers (0 overlaps found). Largely Akkadian — intended cross-script sign-shape transfer, tagged origin=ebl. License not stated upstream → `train_local_only`: these images must not be redistributed with the dataset.

Phase 5 — Packaging (phase5_package.py) ✅

releases/Sumtables-Cuneiform-Full-Fable5-Remaster-v1.0.0: 455,506 records across train.parquet (402,004; 259,457 vision, images at a 2048px training budget — full-resolution originals preserved in Phase1/fetched), validation.parquet (24,507), test.parquet (24,446), and the train-local t5_train.parquet (4,549). Ships MANIFEST.parquet (per-sample provenance for re-weighting without touching image bytes), phase reports, mixer config, dataset card — with every conversation validated against the actual Qwen3-VL chat template (zero flags) and eval↔train tablet disjointness re-verified from the written files at package time. The earlier v0.9.0 pre-release (149,489 records, text-only train) is superseded but retained for provenance.

Phase 6 — Evaluation harness (phase6_eval.py) ✅

Resumable predictions (mock + any OpenAI-compatible server), CER, sign-level F1, chrF, metadata accuracy, abstention precision/recall, insertion-rate honesty proxy; strata by task × period × genre × tier; dual reporting with and without formula-leaked tablets. Validated end-to-end with a floor baseline (all metrics at expected floor over 16,463 real test items).

3. Measured baseline (zero-shot Qwen3-VL-8B-Instruct)

2,000-item random sample, locally served model, pre-v1.0 test parquet (definitive run re-executes on v1.0):

Interpretation: the pretrained model can neither read cuneiform images nor map signs to readings — and it already knows to refuse. Post-training gains on sign-F1/CER are therefore attributable to this dataset, and the training risk to manage is residual over-refusal, not hallucination alone.

4. Training path (first fine-tune: Qwen3-VL-4B)

Two equivalent routes onto the same Unsloth engine — both consume the curated starter pack (4,850 records: 1,500 surface OCR / 1,000 lineart / 500 full-tablet / 400 sign grounding / 250 photo-lineart pairs / 250 metadata / 100 refusal-capped abstention / 850 text tasks; tier-A/B only for OCR; one record per tablet per task):

• —Unsloth Studio (GUI): load the exported dataset directory unsloth_starter/ (export_unsloth.py converts our parquet into the documented Studio format — a messages column with typed content parts and an images column; 4,850 samples, 4,000 with images). Studio's dataset preview should auto-map both columns.
• —Script (`train_lora.py`): the same Unsloth FastVisionModel workflow headless, with the capability guardrails executed automatically — vision-native loader asserted at load, adapter-only artifact, stray-config removal, and verify_model_capabilities.py run on the output (non-zero exit on any regression).

Either way, acceptance is gated (see TRAINING_GUARDRAILS.md, written after a prior non-pipeline fine-tune silently lost vision and clamped context 256k→64k): the served result must pass the live vision probe and the ~70k-token needle probe, then phase6_eval.py on validation against the zero-shot reference (sign-F1 ≈ 0, 93–99% refusal). The 4B run validates the data recipe cheaply; the identical recipe scales to Qwen3-VL-8B-Instruct (--model Qwen/Qwen3-VL-8B-Instruct).

5. Engineering log (what broke and what it taught)

6. Dataset structure

Each record: task, tablet (canonical P-number or ebl: namespace), split, tier, conversations (system/user/assistant JSON, image flag on the user turn, canonical output format per task), image (struct{bytes,path} or null), provenance (JSON: image key, bbox, prompt id, origin, license posture).

Output conventions: transliterations preserve <SURFACE> / <COLUMN> / <RULING> / <BLANK_SPACE>, subscript numerals, damage tokens (<unk>, ...); refusals are exactly <ILLEGIBLE_IMAGE> …; T11 targets are {"period": ..., "genre": ...}; T5 targets are JSON with 0–1000 boxes.

7. Validate it yourself

pip install -r requirements.txt
python -m pytest tests -q                 # 65 tests: split integrity, segmentation,
                                          # grading, alignment, mixer, metrics, leak guards

• —Split integrity: re-run clean_sumtablets.py --dry-run and diff split_manifest.csv; every hard assertion is recomputed from the written parquet, not trusted from memory.
• —Segmentation quality: open Phase1/review_sample.html — 200 seeded images with detected boxes drawn; document-flagged cards are marked.
• —Mixture & provenance: MANIFEST.parquet + build_report.json expose every record's task, source, prompt id, and the mixer's target-vs-achieved shares including underfills.
• —Metrics: phase6_eval.py is deterministic given a predictions file; floor and oracle mock modes bound every metric.
• —Capability preservation (lessons learned from a prior training attempt that silently lost vision and clamped context 256k→64k): every training artifact must pass verify_model_capabilities.py — static checks on architecture, max_position_embeddings=262144, mRoPE, vision tower tensors, processor files, GGUF mmproj, plus live vision and ~70k-token needle probes. Rules and acceptance checklist: TRAINING_GUARDRAILS.md.

8. Status & roadmap

• —✅ v1.0.0 released: CDLI fetch complete (all 178,970 pairs resolved; 71,852 full-res images), full-corpus Phase 1 manifests (126,282 images, 87,764 surface pairs), train vision tasks materialized at a 2048px training budget, floor baseline recorded on the v1.0 test split, tablet disjointness re-verified across all release files.
• —▶ Next: first fine-tune (Qwen3-VL-4B on the starter pack via train_lora.py), gated by verify_model_capabilities.py.
• —⏳ Gold-997 per-item expert verification.
• —Future: ORACC etcsri HTML translation scrape; ETCSL; Q-composite re-admission with leak screening; DeepScribe/MaiCuBeDa ingestion; synthetic font renders (T12).

9. Licensing

10. Acknowledgements

Built on the work of the CDLI, ORACC/ePSD2, ETCSL, and eBL projects and the SumTablets authors (Simmons et al. 2024), representing decades of Assyriological digitization. Dataset restructuring executed with Claude Fable 5 (Anthropic); human direction, review labels, and gate approvals by the project owner.

11. Why Qwen3-VL?

Because the model family is specifically positioned around stronger OCR, rare/ancient character handling, blur/tilt robustness, and long-document structure parsing. That matters for cuneiform because the problem is not only “read image text”; it is damaged visual signs, surface structure, line order, uncertain readings, and translation context.