Keler-Health/turkish-medical-deid-eval

Turkish Medical De-Identification Evaluation Corpus (Synthetic)

A labelled benchmark for evaluating the removal of personally identifiable information (PII) from Turkish medical speech-to-text (STT) transcripts.

This dataset contains no real data. Every consultation, name, phone number, address, identifier and financial detail is programmatically generated and fictitious. The corpus exists specifically so that de-identification systems can be evaluated without any real patient or personal data.

Overview

De-identification systems for medical scribes are difficult to evaluate, because they cannot ethically be tested on real patient data. This corpus addresses that problem: it injects realistic-but-fictitious PII, at known positions, into clean PII-free Turkish medical dialogues. The ground-truth labels are therefore exact and complete by construction, which makes recall — did the system catch every identifier? — directly measurable.

The corpus is deliberately adversarial. It stresses the two forms that matter most in dictated medical speech:

• —Spoken-form identifiers — numbers and addresses spelled out in words (a phone number dictated digit by digit, an e-mail read as "name dot surname at example dot com"). This is the dominant form an STT engine produces and the hardest for digit-based redaction to catch. It accounts for roughly half of all labels.
• —Clinical distractors — clinically legitimate content that resembles PII (medication doses, vital signs, dates, hospital names) and must not be redacted, so that precision and clinical-data preservation can be measured alongside recall.

Files

from datasets import load_dataset

main       = load_dataset("Keler-Health/turkish-medical-deid-eval")                 # default = "main"
atypical   = load_dataset("Keler-Health/turkish-medical-deid-eval", "atypical")
asrnoise_b = load_dataset("Keler-Health/turkish-medical-deid-eval", "asrnoise_b")

Composition

The main config is the reference benchmark. The atypical and asrnoise_* configs are targeted probes derived from the same generator and privacy model; they are meant to be reported alongside main, not in place of it.

`main` — PII categories and surface forms:

Surface forms: spoken (word) 1,239 · plain text 884 · written (digit) 308. Case subsets: standard 300 · confusable-heavy 40 · pure-negative 40 · long 20.

Configurations

atypical — name generalization

In main, person names are drawn from common Turkish name lists, so a system whose own name resources overlap that pool reports optimistic PERSON recall (≈99.7 % of gold name tokens fall in a standard gazetteer). The atypical config isolates generalization: its 82 person names are deliberately out-of-distribution — Levantine-Arabic, Sub-Saharan African, East / South / Central Asian, European, rare-Turkish, and homographic-Turkish names (including everyday words such as "Demir Kaya"). Only ≈19.6 % overlap a standard gazetteer, so the config measures whether a system detects names it has not memorized. Records carry an additional name_profile: "atypical" field.

asrnoise_a / asrnoise_a50 / asrnoise_b — ASR robustness

Real STT output is garbled: a Turkish ASR model mis-writes characters, drops diacritics, truncates suffixes, and deletes or inserts tokens. A system that scores well on clean text can therefore leak PII on real transcripts. These configs measure that gap objectively.

The corruption is anchored to the deployed model's own statistics, not invented. The production Turkish medical Whisper model was run over held-out ASR validation audio, and its error behaviour — per-token-class substitution / deletion / split / join rates, a character-confusion matrix, and an insertion lexicon — was measured into an aggregate error channel (asr_error_channel.json). A span-aware, seeded corruptor then applied that channel to the clean main corpus. The channel was measured on audio fully disjoint from this corpus, so there is no leakage into, or tuning against, the evaluation set.

Corruption invariants (so every scorer still applies): a garbled gold anchor is still gold (a leaked "Ayşse Kayya" is a leak); the gold count is constant (deleting an entire anchor is forbidden); digit-form anchors are left untouched; and the pre-corruption value is preserved in corruption.clean_anchor. case_id is preserved so clean↔noisy pairs align 1:1.

blind_garbled_pii — corpus-blind robustness probe

A 66-row probe with its own schema, authored by annotators who saw only the ASR error taxonomy, never this corpus. Each row is labelled expected: MASK | KEEP:

• —MASK (32 rows) — garbled PII that a robust system should still catch.
• —KEEP (34 rows) — garbled clinical text that must not be over-masked (e.g. "b on eki" = vitamin B12, "Kalb" = kalp / heart).

It measures garbled-PII recall and garbled-clinical precision independently of the main corpus.

asr_error_channel.json

The aggregate error channel from which the ASR corruption was derived — provenance metadata, per-token-class operation rates and subtype distributions, the character-confusion matrix, split statistics, and the insertion lexicon. Published as a plain file (not a loadable config) for full methodological transparency. It contains only aggregate statistics and fragmentary word pairs; no source transcriptions are included (withheld for source-audio licensing reasons).

Data schema

Case configs (main, atypical, asrnoise_*)

One JSON object per line:

{
  "case_id": "deid_eval_v1_0000",
  "subset": "standard",               // standard | long | confusable_heavy | pure_negative
  "num_lines": 30,
  "lines": [                          // the diarized transcript
    {"id": 1, "speaker": "SPEAKER_00", "role": "Doktor", "text": "..."}
  ],
  "transcript": "1| SPEAKER_00: ...", // numbered rendering used for scoring
  "gold_entities": [                  // the ground truth
    {
      "label": "PERSON",              // one of the 6 categories above
      "ids": [2],                     // line id(s) the span occurs on
      "anchor": "Bireylül Ardıç",     // the exact text to be de-identified
      "surface_form": "n/a",          // word (spoken) | digit | n/a (plain)
      "template_id": "person_relative",
      "anchor_context": "clean"       // clean | suffixed
    }
  ],
  "confusables": [                    // clinical distractors that must NOT be labelled
    {"line_id": 6, "class": "dosage_vitals", "text": "beş yüz miligram"}
  ]
}

asrnoise_* records additionally carry a corruption block on garbled entities (clean_anchor, ops) and a case-level corruption stamp (channel_sha256, variant, intensity).

Blind config (blind_garbled_pii)

{
  "text": "b on eki değerim düşük",   // the garbled line
  "span": "b on eki",                 // the span in question
  "label": "FINANCIAL | CONTACT | ... | CLINICAL",
  "expected": "KEEP",                 // MASK | KEEP
  "source": "…", "note": "…"          // provenance / annotator note
}

Suggested evaluation metrics

• —Residual Identifier Rate (primary safety metric) — the fraction of gold_entities whose anchor still appears verbatim on its line after masking. Lower is safer.
• —Recall / precision / F1 against gold_entities, per category.
• —Over-redaction / clinical preservation — how often a confusables span is masked (it should not be).

Intended use and limitations

• —Intended use: benchmarking PII detection and de-identification / redaction systems for Turkish medical STT text, and research on privacy-preserving clinical NLP.
• —Not for training identity models: all names and values are fictitious.
• —Synthetic caveat: the data follows defined rendering conventions and may not reproduce every characteristic of real ASR output. Treat clean-corpus results as an upper bound and validate on real-audio output where possible; the asrnoise_* configs narrow, but do not close, this gap.
• —Gazetteer overlap: in main, person and place values come from curated Turkish lists, so systems relying on the same lists report optimistic recall on unseen entities. Use atypical to measure out-of-distribution name recall.

License

Released under CC BY-NC 4.0 (Creative Commons Attribution-NonCommercial 4.0 International). You may share and adapt the corpus for non-commercial purposes with attribution; commercial use is not permitted. Because the data is fully synthetic it contains no personal data — the non-commercial term is the only condition on use.

Citation

If you use this corpus, please cite it as the Turkish Medical De-Identification Evaluation Corpus (Synthetic) and link back to this repository: https://huggingface.co/datasets/Keler-Health/turkish-medical-deid-eval