stighellemans/meddeid-english-synthetic-benchmark

MedDeID English synthetic clinical benchmark

This is a fixed, human-validated benchmark containing 300 synthetic English clinical documents: 150 en-GB and 150 en-US. It contains 1,717 primary PII spans and 7,358 confirmed core-PII subannotation segments. It contains no real patient notes or personal information.

Use the entire test split only for final evaluation:

from datasets import load_dataset

benchmark = load_dataset(
    "stighellemans/meddeid-english-synthetic-benchmark",
    split="test",
)

Do not use the benchmark for training, validation, early stopping, hyperparameter or threshold selection, prompt or rule development, or choosing among candidate models whose final performance will be reported on it.

Human review and subannotations

Every document and every identifying span in this benchmark was reviewed by a human. After the main spans were finalised, their contents were reviewed a second time at a finer level and divided into 7,358 subannotations.

Subannotations describe what is inside a larger identifying span. A name can be split into given name, family name, initials and formatting; an address into street, house number, municipality and postcode; and a date into day, month and year. Every character inside a reviewed span is accounted for.

This finer layer helps answer a question that ordinary span-level evaluation cannot: did a system remove the identifying content itself while preserving as much surrounding non-identifying text as possible? It can distinguish a missed surname from harmless punctuation, and an exact redaction from one that removes additional clinical context. In the dataset, these pieces are stored under the relevant primary span as subannotations.

Models evaluated on this benchmark

The complete manuscript comparison is shown below. Recall is calculated over the identifying annotation characters. The metadata-enabled condition applies the same optional patient/caregiver-name recovery step to every system. Values in parentheses are document-clustered 95% bootstrap confidence intervals. Higher recall and lower non-PII redaction are better.

These results use the fixed human-validated-v1 benchmark release. The meddeid-english-synth result is an in-domain synthetic pipeline check, not external or clinical validation.

Intended use and limitations

This benchmark supports reproducible comparison of English clinical-text de-identification systems. Its two locale profiles permit separate reporting for UK- and US-style synthetic contexts. Synthetic-benchmark performance must not be interpreted as performance on real clinical notes or as proof that output is anonymous. Deployments require representative local validation.

Annotation guideline

See the English annotation guideline used for the human review and subannotations in this benchmark.

Project and acknowledgements

Developed by Stig Hellemans, Tom Stroobants, Elyne Scheurwegs, Pieter Meysman, Philippe Jorens, and Kris Laukens at the University of Antwerp and Antwerp University Hospital (UZA), with support from Research Foundation Flanders (FWO), grant 1SA3226N.

We thank the participating general practice and Co-Medic for enabling the primary-care validation reported in the accompanying MedDeID study.

Citation

Please cite both the archived dataset version and the accompanying paper.

Dataset

Hellemans, S., Stroobants, T., Scheurwegs, E., Meysman, P., Jorens, P., and Laukens, K. (2026). MedDeID English synthetic clinical corpus, benchmark and annotation guideline (v3) [Dataset]. Zenodo. https://doi.org/10.5281/zenodo.22689857

MedDeID paper

Hellemans, S., Stroobants, T., Scheurwegs, E., Meysman, P., Jorens, P. G., and Laukens, K. (2026). MedDeID enables locally governed clinical-text de-identification from real or synthetic training data. arXiv:2609.10049.

Licence

The benchmark and included guideline are licensed under CC BY 4.0.