orgrctera/pii_masking_300k_information_extraction

PII Masking 300k — Information Extraction

Dataset summary

This repository hosts a validation sample of the PII Masking 300k benchmark for the information extraction track: models must identify personally identifiable information (PII) in text and produce structured extractions (slot-filling JSON), optional token-level BIO labels, and span-based annotations for masking or redaction workflows.

The full PII Masking 300k suite is designed to stress-test privacy-preserving NLP: diverse entity types, realistic surface forms, and multi-field documents. The artifact published here corresponds to a Langfuse-exported split focused on English validation examples.

Current split: validation_sample_200.english — 200 labeled items (see Dataset structure below).

Background

PII masking (also called de-identification or pseudonymization in regulated settings) is the task of detecting sensitive spans in free text—names, contact details, government identifiers, locations, and similar fields—and replacing or suppressing them before logs, datasets, or model prompts leave a trusted boundary.

In practice this is closely related to:

• —Named entity recognition (NER) with a rich label set tailored to privacy (not only “person/location/org”).
• —Structured information extraction (IE), where the output is not only spans but typed slots (e.g., email addresses under EMAIL, national/social numbers under SOCIALNUMBER).
• —Sequence labeling for token classification (e.g., BIO tags aligned with a subword tokenizer), when training discriminative taggers.

Public benchmarks and corpora for clinical de-identification (e.g., i2b2 shared tasks) and open multilingual PII collections (e.g., community “PII masking” datasets on Hugging Face) have driven progress on span detection and replacement for LLM safety and compliance use cases.

Task definition: Information extraction / PII masking

For each item, the target behavior is:

1. 1.Structured IE: Fill a fixed JSON schema of PII slots (many possible keys; absent types are null, present values are typically arrays of strings).
2. 2.Span awareness: Align extractions with character spans in the source text (privacy_mask, span_labels) and optionally masked renderings (target_text with placeholders such as [[EMAIL]]).
3. 3.Optional token-level supervision: Some rows include BIO-style labels and tokenizer-aligned token lists for training or evaluating token classification models.

The column input.keys lists the PII types relevant to that example (space-separated), which can be used to scope evaluation or multi-task training.

Representative slot names in the schema include (non-exhaustive): EMAIL, TEL, SOCIALNUMBER, DRIVERLICENSE, DATE, TIME, STREET, CITY, STATE, COUNTRY, POSTCODE, USERNAME, GIVENNAME1, GIVENNAME2, LASTNAME1, LASTNAME2, LASTNAME3, TITLE, SEX, PASSPORT, IDCARD, IP, GEOCOORD, BUILDING, SECADDRESS, BOD, PASS, and others as defined in the benchmark.

Dataset structure

Format: Parquet-backed dataset on the Hub; no custom loading script required.

Examples

Example 1 — Contact details (EMAIL, SOCIALNUMBER)

`input.keys`: EMAIL SOCIALNUMBER

`expected_output` (abbreviated):

{
  "EMAIL": ["24SA@gmail.com"],
  "SOCIALNUMBER": ["742-240-8260"],
  "TEL": null,
  "CITY": null
}

`metadata.item` excerpt: source_text contains the raw sentence with inline values; target_text replaces spans with placeholders such as [[EMAIL]] and [[SOCIALNUMBER]]; privacy_mask lists {start, end, label, value} records for evaluation.

Example 2 — Mixed fields (TIME, DATE, STREET)

`input.keys`: TIME DATE STREET

The source passage describes a scheduled event with a time, calendar date, and street address; gold extractions populate the corresponding slots in the same JSON schema, with spans aligned to the narrative text.

(Full strings are available in the `metadata.item` field for each row.)

Intended use

• —Training or fine-tuning IE / NER models for PII detection and redaction.
• —Evaluation of precision/recall on typed spans and structured outputs against expected_output and span metadata.
• —Benchmarking RAG or LLM pipelines that must refuse or mask sensitive content before indexing or display.

Limitations: This release is a sample split (200 rows), not the full 300k-item corpus. Results on this subset are indicative only; report benchmark name and split when comparing systems.

Ethics and privacy

Synthetic or semi-synthetic text is used to reduce direct re-identification risk. Treat any exported text as sensitive unless your organization has cleared it for public use. Do not use this dataset to attempt to re-identify individuals or to bypass organizational data policies.

References and further reading

1. 1.Clinical de-identification at scale — Beyond Accuracy: Automated De-Identification of Large Real-World Clinical Text Datasets discusses operational de-identification on large clinical corpora and comparisons to commercial APIs; useful context for real-world error rates and deployment concerns. arXiv:2312.08495

1. 1.Survey / systematic view — Reviews of machine learning for clinical text de-identification summarize NER/hybrid pipelines, datasets such as i2b2, and typical PHI categories aligned with regulations. Search: systematic review “de-identification clinical text” / “named entity recognition PHI”.

1. 1.Regulatory framing (US) — HIPAA Safe Harbor lists categories of direct identifiers (e.g., names, geographic subdivisions smaller than state, dates, telephone/fax, email, SSN, medical record numbers, etc.) commonly used as checklists for de-identification (informative; not legal advice).

1. 1.Open PII-masking resources — Community datasets such as ai4privacy multilingual PII masking collections on Hugging Face illustrate large-scale span-annotated resources for assistant and LLM safety; terminology and label taxonomies are adjacent to this benchmark track.

1. 1.NER foundations — Classic sequence labeling for NER (CRF, BiLSTM, Transformers) applies directly to token-level PII tagging; structured JSON slot filling can be implemented as constrained decoding, seq2seq, or pipeline (tagger + normalization).

Citation

If you use this dataset, cite the PII Masking 300k benchmark and this Hub repository, for example:

@dataset{pii_masking_300k_information_extraction,
  title = {PII Masking 300k: Information Extraction (validation sample)},
  author = {CTERA AI},
  year = {2025},
  howpublished = {\url{https://huggingface.co/datasets/orgrctera/pii_masking_300k_information_extraction}}
}

Adjust author/year once your canonical paper or tech report is available.

Provenance

• —Benchmark: pii_masking_300k
• —Track: information_extraction
• —Export tooling: Langfuse export to Hugging Face Datasets

For questions about the full benchmark or additional splits, contact the dataset maintainers via the Hugging Face community tab.