NikolaiSachok/strata-insurance-corpus

Strata Insurance Corpus

A reproducible, fully synthetic, multi-format insurance document corpus for a fictional pan-European property-&-casualty insurer, Meridian Mutual, shipped with a golden evaluation set produced by construction. Built to exercise and benchmark document-RAG systems on enterprise-shaped data — born-digital and scanned PDFs, Word documents, spreadsheets, and photos — with trustworthy ground truth.

Everything here is synthetic. No real persons, companies, or policies; any resemblance is coincidental. Documents are marked synthetic in metadata and, where visible, on the page. The realistic synthetic PII (names, addresses, national IDs, plates, faces) is intentional redaction-test material — handling it is the consuming RAG layer's job, not this corpus's.

Generated by a seeded, deterministic pipeline (MIT-licensed): [github.com/NikolaiSachok/strata-insurance-corpus](https://github.com/NikolaiSachok/strata-insurance-corpus). Composes with the Strata-RAG engine, but is engine-agnostic — this is data + ground truth only; the consuming system owns ingestion.

What's inside

• —305 entities (policyholders / policies / claims / agents / adjusters) across six Eurozone countries (DE/FR/ES/IT/NL/IE), € amounts, DD/MM/YYYY dates.
• —1,311 documents: 753 PDF (born-digital declarations, contracts, FNOLs, adjuster reports, estimates, settlement/denial letters, ID cards) · 121 Word .docx · 3 Excel · 1 CSV · 2 Markdown · 431 JPG — 271 scanned (OCR-target) variants incl. 27 scan-only police reports, and 160 AI images (80 damage/evidence photos + 80 synthetic ID portraits).
• —671 golden questions — 444 text (361 semantic + 3 aggregation + 80 multi-hop) + 227 multimodal (27 OCR + 80 vision + 80 multimodal-retrieval + 40 cross-modal).
• —6,543 PII spans catalogued for redaction scoring.
• —Demographically diverse synthetic faces. Policyholder gender is modelled (name ↔ portrait ↔ passport MRZ agree), and the 80 ID portraits span an even spread of ethnic appearance (skin tone, features, hair). This is deliberate: face-detection / redaction / vision models have documented demographic bias, so a homogeneous face set would hide it — diverse faces let a consuming system measure per-group performance.

Files

You own ingestion

The corpus ships documents in their native forms and does not pre-extract a clean-text layer — that's a design choice, not an omission. Parse PDFs/docx/xlsx with your own stack; OCR the scanned JPGs (shipping their clean text would hand you the answer and defeat the test); understand the photos with your own vision/caption/embedding. This is what keeps it a genuine benchmark rather than a solved text dump.

Ground truth — golden.jsonl

One JSON object per line, aligned with general enterprise-RAG benchmark formats:

{"id": "Q-KB-lines", "question": "Which lines of business does Meridian Mutual underwrite?",
 "answer": "Motor, Household, Commercial", "relevant_doc_ids": ["DOC-KB-UW"],
 "query_class": "semantic", "modality": "text", "provenance": {"entity_id": "CORPUS", "field": "lines_of_business"}}

Every answer is grounded in document provenance — it is exactly what its cited documents state, and relevant_doc_ids is exactly the set that asserts it. The multimodal classes have answers that live only in a scanned or image document (the seeded image prompt-spec / rendered scan is the label), and a leak-guard guarantees each OCR/vision answer appears on no born-digital page — so those questions truly require the modality.

How to score

A dependency-free reference scorer ships in the generator repo so everyone scores identically:

python -m generator.eval --golden golden.jsonl --predictions your_preds.jsonl
# Recall@K / nDCG@K / exact-match / token-F1, broken down by query_class AND modality

Predictions are JSONL: {"id": "...", "retrieved_doc_ids": ["DOC-…", …], "answer": "..."}.

License & citation

Data: CC-BY-4.0 (use/share/adapt freely, including commercially — just credit the source). Generator code: MIT. All data is synthetic.

@misc{strata_insurance_corpus,
  title  = {Strata Insurance Corpus: a synthetic multi-format document-RAG benchmark},
  author = {Sachok, Nikolai},
  year   = {2026},
  url    = {https://github.com/NikolaiSachok/strata-insurance-corpus}
}