stighellemans/meddeid-dutch-synthetic-benchmark

MedDeID Dutch synthetic benchmark

This repository contains the fixed 300-document synthetic Dutch evaluation benchmark. It contains no real patient notes and must not be mixed into a training or validation partition when reporting MedDeID benchmark results.

This is the openly shareable synthetic benchmark described in the manuscript. It is not the separate 300-note hospital benchmark, which contains personal information and is not publicly distributed.

Subannotations and human review

Every document, primary annotation and subannotation in this benchmark was reviewed by a physician-researcher. Primary annotations mark the complete text that a de-identification system may need to remove. Subannotations divide that span into the identifying core and surrounding formatting or clinical context.

This distinction addresses an important limitation of all-or-nothing span scoring: a system may find the actual identifier while trimming punctuation or titles differently, or it may remove extra non-identifying words around it. Subannotations make it possible to measure both whether the identifying content was recovered (core PII recall) and how much unrelated content was removed (non-PII redaction). The 300 documents contain 3,598 primary spans and 14,591 reviewed subannotation segments.

Annotation guideline

See the Dutch annotation guideline.

Models evaluated on this benchmark

These are the synthetic-benchmark columns from the comparison table in the accompanying manuscript.

¹ The hospital-trained weights are not publicly distributed; the link points to the MedDeID software repository used to run the model.

Project and institutional context

Developed by Stig Hellemans, Tom Stroobants, Elyne Scheurwegs, Pieter Meysman, Philippe Jorens and Kris Laukens at the University of Antwerp and Antwerp University Hospital (UZA). This work is supported by a Research Foundation Flanders (FWO) SB Fellowship, grant 1SA3226N.

We thank Co-Medic for supporting the project.

Author ORCIDs: Stig Hellemans, Tom Stroobants, Elyne Scheurwegs, Pieter Meysman, Philippe Jorens, and Kris Laukens.

Citation

Please cite both the archived dataset release and the accompanying paper.

@dataset{hellemans_2026_meddeid_dutch_data,
  author    = {Hellemans, Stig and Stroobants, Tom and Scheurwegs, Elyne and Meysman, Pieter and Jorens, Philippe and Laukens, Kris},
  title     = {MedDeID Dutch synthetic corpus and benchmark},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.22689856},
  url       = {https://doi.org/10.5281/zenodo.22689856}
}

@misc{hellemans2026meddeid,
  author        = {Hellemans, Stig and Stroobants, Tom and Scheurwegs, Elyne and Meysman, Pieter and Jorens, Philippe G. and Laukens, Kris},
  title         = {{MedDeID} enables locally governed clinical-text de-identification from real or synthetic training data},
  year          = {2026},
  eprint        = {2609.10049},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2609.10049}
}

When reporting results, also record the immutable dataset revision, model revision, inference settings, and meddeid-eval version.

Licence

The synthetic benchmark and included annotation guidelines are licensed under the Creative Commons Attribution 4.0 International licence (CC BY 4.0). Cite the MedDeID dataset when using this resource.