Marawanelbalal/synthetic-pii-phi-v2

Synthetic PII/PHI v3

Synthetic clinical/administrative documents in three locales (en-GB, nl-BE, fr-FR) with span-level PII/PHI annotations, built for training a multilingual de-identification (token classification) model. Every entity value — names, IDs, dates, addresses, lab results — is synthetically generated. No real patient, provider, or organization data appears anywhere in this dataset; city names and their postcode stems are real (a city name is not personal data and a document needs a plausible place), everything else is constructed.

This repository is versioned as v2 on the Hub for continuity, but the content below is the v3 build — see "What changed in v3" for what distinguishes it from the original v2 release.

Splits

What changed in v3

1. Terminology-pool separation for DIAGNOSIS / SYMPTOM / LAB_RESULT. The v2 release drew DIAGNOSIS and SYMPTOM values from the same shared terminology pool, and LABRESULT mixed genuine quantitative lab/staging concepts with patient-reported symptom findings (e.g. SNOMED 29303009 "dyspnoea"). Verified directly against the published v2 data: DIAGNOSIS and SYMPTOM values overlapped on 92% of SYMPTOM's distinct terms, and a cluster of social-determinant-of-health administrative concepts (food insecurity, homelessness, "no fixed abode", self-harm, social isolation) leaked into DIAGNOSIS, SYMPTOM, and LABRESULT spans. v3 assigns each of these three labels its own dedicated, code-reviewed terminology pool (condition for DIAGNOSIS, a newly-split-out symptom pool for SYMPTOM, observation for LABRESULT minus symptom/duplicate codes), and drops SDOH concepts from all three entirely — measured post-fix: DIAGNOSIS/SYMPTOM overlap is 6 terms out of 1,687 (a residual set of legitimately dual-use terms like "chronic pain", not a pool-sharing defect), LABRESULT/SYMPTOM overlap is 0.

2. Lower single-wording reuse. Measured directly on this release: the most-reused exact document text appears in at most 0.0205% of documents (down from v2's already-low ~0.10-0.11%, down from this release's own 0.041% pre-fix figure before point 4's patch, and down again from 0.031% after point 4's patch but before point 5's — the identifier-reformat, postcode, phone, and decoy-append patches all individually rewrite text, so each pass further reduces exact-duplicate document text as a side effect), reducing sentence-level memorization risk further.

3. Row counts and split proportions differ from v2 (48,783 total here vs. v2's 47,153) because this is a fresh build from the same generation pipeline at a later point, not a patch applied to the identical v2 document set — every row that appears here was independently rendered.

4. Identifier surface-form diversity and a hard-negative decoy, applied as a targeted post-render patch. Two defects surfaced while evaluating a model trained on the pre-patch build:

• —National/provider ID values were always rendered bare (e.g. 70111915942), because the value-generation step never called the existing surface-form renderer. Real documents print these digits with the separators their issuing system actually uses (grouped, dotted, or dashed), so a model only ever saw the unseparated form and missed the common ones at evaluation time. NHS_NUMBER, GMC, INSZ, RIZIV, NIR, and RPPS values are now reformatted into their real-world surface forms; the same underlying value gets the same formatting everywhere it recurs within one document. HEALTH_INSURANCE_ID has no defined surface forms in any locale and is unaffected (bare digits are its only correct form).
• —No BMI-shaped hard negative existed in training, so a model had never seen a plausible-looking decimal measurement it should leave unmasked, and produced false positives on BMI-shaped numbers at evaluation time. ~20% of documents now carry one appended hard-negative sentence (a calibration reading, an audit reference, a stock count) containing a BMI-shaped decimal value that is deliberately not annotated as an entity.

Both fixes were applied as a post-render patch directly on the already-final, already-verified rendered documents — splicing the new identifier text in place and shifting only the offsets of entities after it, and appending the decoy sentence after the last entity — rather than by re-running composition or realization selection. This guarantees the set of realizations used, and therefore the reuse-share figure above, is unaffected by construction: no composition, blueprint, or candidate-wording selection was touched, only the value layer and a trailing sentence.

5. A second post-render patch, root-caused directly from a trained model's per-label weaknesses on GMC, PHONE_NUMBER, BUILDINGNUM, and ZIPCODE. Investigating those four labels' false positives/negatives found one genuine annotation defect and one training-data blind spot:

• —en-GB `ZIPCODE` values were outcode-only (e.g. M9, never M9 4TF). A real UK postcode is an outward code (the district — the only part that has a reference list) plus an inward code (a digit and two letters, assigned per-address, which is why no reference list of it exists or should) — real letters always print both halves space-separated. A model trained exclusively on the outward half learned that a postcode is the outward half, and at evaluation time on genuine two-part postcodes it segmented the two apart. Every en-GB ZIPCODE value is now completed with a synthesized inward code (excluding the letters C, I, K, M, O, V, matching Royal Mail's PAF specification) — 1,129 values completed across the corpus.
• —Phone numbers were always written in national form, never internationally, even though real reference letters write a meaningful share of numbers with a country-code prefix. ~40% of PHONE_NUMBER values are now rewritten into one of four international forms (e.g. +32(0)32 0986573, +3254 7336249), dropping the national trunk zero and prepending the country code — the same value gets the same rewrite everywhere it recurs within one document — 4,543 values rewritten.
• —Decoy blindness. The held-out benchmark used to evaluate this corpus's models tests against a fixed, repeating set of hard-negative shapes (a lab-request number, a device serial, a storage-unit code, a billing/administrative code pair, a room/bed transfer note) that this training corpus had no equivalent of at all, so a model pattern-matched on digit shape alone at high confidence and produced false positives on GMC/PHONE_NUMBER/ZIPCODE/BUILDINGNUM-shaped numbers it had never been taught to leave unmasked. Five new hard-negative sentence families, each independently rolled at ~15% per document with freshly authored wording (not copied from any evaluation fixture), are now appended per locale — 27,140 decoy sentences appended across the corpus, on top of point 4's BMI decoy.

Like point 4, all of point 5's fixes are post-render patches on the same already-final rendered documents — entity-splicing for the postcode/phone fixes (same offset-preserving splice-and-shift technique as the identifier reformat), append-only for the new decoys — so realization selection is again untouched and the reuse-share figure above already reflects this pass.

6. Ordinal-suffix and month-first DATE surface forms, applied as a third post-render patch. Every DATE value in this corpus, before this patch, was either fully numeric (18/03/2010) or plain cardinal-spelled (18 March 2010) — no locale ever produced an ordinal suffix, a month-first ordering, or a fully spelled-out word-form day number. A model trained exclusively on those two shapes has no reason to recognise 18th March 2010, 18th of March 2010, March 18th, 2010, or the eighteenth of March 2010 as dates rather than arbitrary text — a real gap if the deployed de-identifier's regex layer misses one of those shapes and the model is the only remaining line of defence. tools.locale_identifiers.generate_date now includes these as a low-weight minority for future builds (en-GB gets all three ordinal/month-first forms, nl-BE gets the 18e maart 2007 abbreviation, fr-FR only ordinalizes the first of the month — 1er mars, never 2e mars — because French genuinely has no ordinal beyond that), and this corpus's existing DATE entities were patched to match: 10,022 of 139,408 DATE entities (7.2%) were reformatted into one of these shapes, via the same offset-preserving entity-splice technique as points 4 and 5, so realization selection and the reuse-share figure are again unaffected. The fully spelled word-form (the eighteenth of March 2010) is deliberately not part of generate_date's output — it exists only in the held-out clinical_letter_eval benchmark, as a shape the training corpus is expected to generalise to rather than have memorised outright.

Everything else — taxonomy, blueprint/composition/rendering methodology, leak-safe splitting, entity-free documents, abbreviation pass, known narrative-contradiction limitation — is unchanged from v2 and documented in full below.

Methodology, start to finish

1. 1.Taxonomy. 39 entity labels (resources/taxonomy/multilingual_taxonomy.json), including per-locale national-ID and provider-ID schemes with their own checksum validators (e.g. NHS_NUMBER/GMC for en-GB, INSZ/RIZIV for nl-BE, NIR/RPPS for fr-FR). Only 32 of the 39 labels appear in this release — see "Entity labels" below — because generation is scoped to 3 of the taxonomy's 6 supported locales (see "Locale scope").

1. 1.Blueprint generation. An LLM writes short text templates ("blueprints") per document family and locale, each a clause with {{PLACEHOLDER}} slots tagged against the taxonomy. Blueprints are independently reviewed and validated (clause-count, placeholder-compatibility, and required/optional placeholder rules) before acceptance. Blueprints also carry a role (e.g. an admission section can be authored as a "positive", "negative", or "resolved" variant of the same clinical fact) so a document family can assemble different narrative arcs from the same slot.

1. 1.Value/identity population. For each rendered document, one synthetic identity and one set of entity values (name, dates, IDs, measurements, diagnoses, medications, ...) is sampled once and shared across every section of that document, so a patient's name, MRN, and dates stay consistent throughout — never resampled per section. DIAGNOSIS, SYMPTOM, and LAB_RESULT values are sampled from three separate, non-overlapping terminology pools (see "What changed in v3").

1. 1.Composition. document_family_matrix.json / document_family_slots.json define which blueprint slots make up each document family (e.g. an admission note has a presenting-complaint slot, a vitals slot, a diagnosis slot, a plan slot). build_document_compositions.py --variants N controls how many distinct slot-arrangements ("compositions") are built per family/locale — more variants means more distinct document shapes, not more copies of the same shape.

1. 1.Rendering. Each composition is rendered --fills N times, each fill drawing a fresh synthetic identity and a fresh candidate wording per section (round-robin over that section's candidate pool, so reuse is spread evenly rather than clustering on whichever candidate sorts first). Post-render, a handful of deterministic, entity-offset-safe cleanup passes run: collapsing template/value collisions ("94 kg kg" → "94 kg", "Easton Royal Royal Infirmary" → fixed at the generator level, "no no fixed abode" → "no fixed abode"), and capitalizing any paragraph that opens with a lowercase placeholder-derived word.

1. 1.Deterministic abbreviation pass. ~15% of the full candidate wording pool (sampled once, all roles/locales) has locale-specific clinical shorthand applied mechanically — no LLM call, just phrase substitution (e.g. "patient" → "pt", "diagnosis" → "dx" in en-GB; "patiënt" → "pt" in nl-BE; "patient" → "pt" in fr-FR) — so the corpus isn't uniformly written in full clinical prose. Every substitution is re-validated against the same structural validator used at generation time; anything that fails validation (e.g. drops a candidate below its minimum word count) is reverted rather than forced through.

1. 1.Entity-free documents. A separate, smaller batch of compositions is built to contain no PII/PHI entities at all (e.g. a generic administrative notice with no patient-identifying content), so a de-identification model also sees genuine negative examples during training instead of learning that every document contains something to mask. These compositions are included in the same leak-safe clustering as the main corpus (see below), and in this build they all happen to cluster into train — none land in development/test. That's an accepted outcome, not a bug: precision on entity-free-style text is still exercised through the model's overall false-positive rate on development/test, most of which is prose with sparse entity density.

1. 1.Splits. Leak-safe: every composition (both the main and entity-free corpora) is clustered by shared blueprint_id, and each cluster — however many documents it spans — is assigned to exactly one split, so no template's exact wording appears in more than one split.

1. 1.Verification. Before release: single-wording-realization reuse share sits at ≤0.0205% per locale (at most a handful of documents anywhere in a locale share the exact same section wording), DIAGNOSIS/SYMPTOM/ LAB_RESULT terminology pools are confirmed independently non-overlapping, and both post-render patches (see "What changed in v3", points 4 and 5) were independently verified to introduce zero span/offset defects across all 48,783 documents (no text[start:end] != value mismatches, no overlaps, no non-NFC text) and zero change to row counts.

Variants and fills

These are the two knobs that shape corpus size and repetition, and it's easy to conflate them:

• —`variants` (composition-build time) — how many different arrangements of blueprint sections exist per document family/locale. Higher variants means more distinct document shapes.
• —`fills` (render time) — how many times each of those shapes is rendered with a fresh identity and fresh section wording. Higher fills means more copies of each shape.

Paraphrasing pass (post-v2 addendum, carried into v3)

After the initial v2 release, the highest-reuse blueprints in each locale (the ones a de-identification model would otherwise see with the most repeated exact wording) were sent through an LLM paraphrasing pass: for each of the top ~500 highest-reuse blueprints per locale, an LLM rewrote the existing candidate wording while keeping every {{PLACEHOLDER}} token, its polarity (assertion vs. negation), and its allowed facts unchanged, so the rewrite could be re-validated by the same structural validator used at generation time and safely mixed into that blueprint's candidate pool. Structural validation alone was not sufficient to trust the output — a manual review found fluency defects (pure sentence-reordering with no lexical change, a Belgian Dutch verb-order error, byte-identical no-op rewrites, garbled French negation) that were filtered out by a second LLM fluency judge plus manual spot-checks. Of 4,060 requests: 3,715 passed structural validation, 1,104 survived automated quality filtering, and 794 candidates (en-GB 352, nl-BE 200, fr-FR 242) were accepted into the corpus.

Annotation-boundary correction (2026-08-19, carried into v3)

A full BIO-compatibility scan found six overlapping STREET spans in train. Each span was shortened to the end of its street name. No text, document, label, or other span changed. A complete scan now confirms that every split has non-overlapping spans.

Entity labels

32 of the taxonomy's 39 labels appear in this release (7 are locale-specific identifier schemes tied to locales not included in the 3-locale scope below).

Core (all 3 locales): AGE, BLOODTYPE, BMI, BUILDINGNUM, CITY, DATE, DIAGNOSIS, DOSAGE, EMAIL, GENDER, HEALTHINSURANCEID, HEIGHT, LABRESULT, MEDICATION, MRN, NAME, ORGANIZATIONID, ORGANIZATIONNAME, PHONE_NUMBER, PROCEDURE, SEX, STREET, SYMPTOM, URL, WEIGHT, ZIPCODE.

Locale-specific identifier schemes: NHS_NUMBER, GMC (en-GB); INSZ, RIZIV (nl-BE); NIR, RPPS (fr-FR).

Locale scope

This release covers en-GB, nl-BE, fr-FR only, a deliberate narrowing from the taxonomy's full 6-locale design (which also covers de-DE, es-ES, ro-RO). The other 3 locales' identifier schemes (CNP, DNI/NIE, STEUER_ID/LANR, COLEGIADO, PARAFA) do not appear in this release.

Schema

Each row:

• —source_text (str): the full document text.
• —privacy_mask (list[{start, end, label, value}]): character-offset spans with their entity label and the substring value.
• —language (str): locale code (en-GB, nl-BE, fr-FR).
• —source_dataset (str): provenance tag.
• —dataset_split (str): train / development / test.

Known limitations (carried from v2, unchanged in v3)

• —Within-document narrative contradictions. Because composition combines independently-authored blueprint sections (some written as "positive", "negative", or "resolved" variants of the same clinical fact) that share one document's sampled diagnosis/symptom value, a document can end up asserting and negating the same finding in different sections. Entity spans and labels are unaffected — every mention of the diagnosis, negated or not, is still correctly tagged — so this degrades narrative realism, not annotation correctness.
• —Entity-free documents are train-only. All entity-free compositions land in train under leak-safe clustering; development/test contain none.

Source

Built from a Synthea-derived synthetic clinical document generation pipeline; documents are composed from LLM-generated text templates ("blueprints") filled with synthetically generated identity/entity values.