TonyYun/pii-shield-benchmark

PII Shield Benchmark

Five public PII-detection datasets rewritten into one record shape, so models can be trained and evaluated across all of them without writing five parsers. 2,301,892 records carrying 13,338,839 labeled spans in 25 European languages.

Nothing here is new text. Every document comes unchanged from one of the five source datasets below; this repository harmonizes the containers — one schema, one label taxonomy, one file format (Parquet, zstd) — and publishes a cleaned version of the annotations: the source labels went through a correction pass that relabeled mislabeled spans, fixed wrong boundaries, and removed spans that do not denote PII. The original source label rides along on every span as native_label, so every harmonization or cleaning decision remains visible and reversible.

Record shape

Every config yields records with the same fields:

{
  "uid": "nemotron-pii/train/000042",
  "source": "nemotron-pii",
  "native_split": "train",
  "language": "en",
  "text": "... the raw document text ...",
  "spans": [
    {
      "entity": "PHONE_NUMBER",
      "sub": "fax",
      "start": 118,
      "end": 132,
      "text": "+977 1 4582 3941",
      "native_label": "fax_number"
    }
  ],
  "meta": { "domain": "healthcare", "locale": "us" }
}

• —start / end are Unicode code points into text, end-exclusive. The invariant text[start:end] == span.text holds for every span in the dataset and is verified at build time.
• —entity is one of 47 shared entity types (list below). sub optionally carries a finer role (for example given vs family under PERSON_NAME), and native_label is always the source dataset's own label.
• —native_split is the source's original split name (train/validation/test/dev/all), preserved as data. All records of a config are published under a single train split; carve your own subsets from native_split or uid.
• —meta passes source-specific extras through (domain, locale, document type, template id, and so on); its fields differ per config.

Sources

The two mapa configs are the only human-written text in the pool; everything else is synthetic. mapa ships two incompatible taxonomies at two granularities, which is why it becomes two configs. 7,564 of the mapa sentences contain no PII at all and are kept as negatives for false-positive measurement.

How it was derived

1. 1.Structural examination. Each source stores annotations differently: character spans in JSON lines, python-repr strings inside Parquet, surface strings without any positions, sentence-level and document-level offsets. Each got a dedicated converter.
2. 2.Label unification. The five sources define 163 distinct labels between them. Each was mapped, with per-label evidence from sampled values, onto 47 shared entity types. Merges never destroy information: the finer distinction moves into sub (for example fax_number becomes PHONE_NUMBER with sub: "fax") and native_label keeps the original. Three privy classes were dropped as non-PII payload: O (2,512,258 spans of filler), FINANCIAL (a mixed bag of currency codes and routing-shaped numbers) and CURRENCY (zero spans).
3. 3.Verification before build. Every record of every source file was scanned: every label known, every offset inside bounds, every stored surface equal to the text slice. The scan established, among other things, that all sources use code-point offsets and that every gretel surface occurs exactly once in its text.
4. 4.Conversion with loud failure. The build refuses to continue on an unknown label or an unexplained text mismatch, and the finished output is read back and checked against span counts pinned from the verification scan, so a converter bug cannot pass silently.
5. 5.Annotation cleaning. The unified annotations then went through a reviewed correction pass: mislabeled spans were relabeled, wrong boundaries were fixed, and spans that do not denote PII were removed. The record texts were never touched, and native_label always keeps what the source originally asserted.

What was deliberately not carried over: the sources' masked-text renditions and token-level BIO/BILOU layers (both regenerable from the spans), and privy's small config (the same generator as its large config, re-sampled with abbreviated label names).

The 47 entity types

LOCATION is the coarse geographic class (privy's LOCATION, mapa's country-level ADDRESS); the six fine location entities can be rolled up into it for coarse scoring.

Known limitations

• —All IBANs in the dataset (16,897 spans, all from privy) are GB-prefixed. There are no EU-format IBAN positives.
• —The synthetic sources inherit their generators' quirks. The published annotations have been cleaned to correct such artifacts (see step 5 above), but no claim of completeness is made — residual issues may remain, and spans the sources never annotated in the first place are not added by the cleaning.
• —Rule-of-thumb quality differs by source: mapa is human-annotated real text, everything else is template- or LLM-generated.

Usage

from datasets import load_dataset

ds = load_dataset("TonyYun/pii-shield-benchmark", "openpii-1m", split="train")
r = ds[0]
for s in r["spans"]:
    print(s["entity"], repr(r["text"][s["start"]:s["end"]]))

Each config directory also contains a sample.jsonl with 100 stride-sampled records for quick inspection without loading anything, and manifest.json at the repository root records per-source input/output counts and drops for the build.

Versions

• —v2 (2026-08-12) — cleaned annotations replace the as-shipped source labels (relabels, boundary fixes, non-PII span removals; texts unchanged).
• —v1 (2026-08-07) — initial release carrying the source annotations as-is.

Licensing and attribution

The compilation is published under CC-BY-4.0. Individual configs remain governed by their source licenses listed above; the mapa text originates from EUR-LEX and is reused under the European Commission's reuse policy (Decision 2011/833/EU). Please cite the original datasets when you use the corresponding configs: ai4privacy, NVIDIA, Gretel, the privy generator authors, and the MAPA project.