mapo80/aliasit-pii-dataset-v3

aliasit-pii-dataset-v3

Multilingual PII span-annotation dataset, 79 entity types, 30 pinned sources, Italian-first. Text plus character spans, single-label BIO.

Splits

`benchmark`, `test` must not be used for training, nor for threshold calibration, model selection, distillation or hyper-parameter tuning. Anything that needs data for those uses validation. This is not a formality: these splits are how the project measures whether a model works, and an evaluation that cannot fail is not an evaluation.

Sources and licences

Every source is pinned: a 40-character commit SHA for Hugging Face repositories, a SHA-256 of the downloaded file for public web files. A branch is not a pin.

Audited and mapped but contributing no documents, because a higher-priority source had already covered the need: richardyoung/synthea-575k-patients, itamarcohen/insurance_car_Qfiles, strova-ai/financial_credit_dataset.

Attribution notices

These sources are redistributed under licences that require attribution. If you redistribute this dataset, these notices must travel with it.

• —NVIDIA, Nemotron-PII (CC-BY-4.0)
• —Agenzia Italiana del Farmaco (AIFA), liste dei farmaci di classe A (CC-BY-4.0)
• —Ministero dell'Interno - ANPR, archivio dei comuni (CC-BY-4.0)
• —Ministero dell'Interno - ANPR, tabella 2 stati esteri (CC-BY-4.0)
• —GeoNames (CC-BY-4.0), mirror DataDock/geonames
• —dossier-legal, italian-legal-corpus (CC-BY-4.0)
• —Ethosoft, TR-DocVQA-Synth (CC-BY-4.0)
• —Ministero della Salute, Manuale ICD-9-CM versione italiana 2007 (IODL 2.0)
• —Toridion, Lindisfarne M1 (CC-BY-4.0)
• —IVASS - Istituto per la Vigilanza sulle Assicurazioni, Lista imprese assicurative vigilate (CC-BY-4.0)
• —Ai4Privacy / Ai Suisse SA (CC-BY-4.0)

Licence mix: APACHE-2.0 (10), CC-BY-4.0 (10), CC0-1.0 (1), IODL-2.0 (1), MIT (5). The strictest terms are CC-BY-4.0 and IODL-2.0 — both allow commercial use and derivative works and require attribution — so the dataset as a whole travels as CC-BY-4.0 with attribution.

Record schema

id                  stable document id
text                the document
language            ISO 639-1, or `und`
split               train | validation | test | benchmark
ent_start[]         character offsets, `end` = start + len, exclusive
ent_len[]  ent_label[]  ent_source_label[]  ent_source_dataset[]  ent_mapping[]
unm_*[]             source annotations with no canonical counterpart, kept on the record
prov_*              dataset, repo_id, revision, original_id, licence, register, ...
fp_*                exact, normalized, entity_set, template, template_shape
cont_extra_json     origin (v2 / v3_delta…), source_family, annotation_origin, group_id

Invariants checked at build time and by the gate:

1. 1.text[start:end] == entity text for every entity and every unmapped span;
2. 2.0 <= start < end <= len(text);
3. 3.entities never overlap (the model is single-label BIO);
4. 4.every label exists in the shipped taxonomy;
5. 5.spans survive the round trip through the tokenizer, BIO encoding and decoding.

Known data gaps

2 categories sit below a hard support threshold, 1 of them critical. They are listed rather than hidden, and each says which source was examined and why it was not usable.

Three entity types were removed from the taxonomy rather than filled with the wrong data: serial_number and vehicle_registration_number (no public source annotates them as the taxonomy defines them), and digital_signature (39% of its existing gold was the phrase "digital signature" instead of a value). Their spans were migrated or kept as unmapped spans, never silently dropped.

Limitations

1. 1.Synthetic. Effectively all of it. No real person's data is in here, which is the point, but synthetic-to-real transfer is not measured by anything in this repository.
2. 2.Derived frames. Part of the support comes from documents written around structured records rather than natural text. Every record declares which it is in cont_extra_json.annotation_origin.
3. 3.Partial annotation from one source. E3-JSI/synthetic-multi-pii-ner-v1 has an open type vocabulary and only a verified allow-list is mapped, so its documents carry annotations for some of their entities and not all. For that reason every such document sits in train only.
4. 4.Reference-model contamination. urchade/synthetic-pii-ner-mistral-v1 is the training set of urchade/gliner_multi_pii-v1, and rizzoaiacademy/anonimizzazione-testi-italiano-clean is declared as the training corpus of rizzo-pii-0.3B. Source validation and benchmark are different questions and must not be reported as one.
5. 5.Two known id collisions. nvidia/Nemotron-PII reuses the same uid across rows with different text, so two document ids sit on more than one document, and one value of medical_record_number crosses the train/validation boundary. It is measured, named, and cannot be fixed without rewriting a frozen document.

Verifying what you downloaded

manifest.json ships with the files and carries the SHA-256 of each parquet. A dataset without a committed checksum is not a version, it is a copy — if a hash differs, this card does not describe what you have.

import hashlib, json, pathlib
m = json.load(open('manifest.json'))
for f in m['files']:
    h = hashlib.sha256(pathlib.Path('data', f['path'].split('.')[-2] + '.parquet').read_bytes()).hexdigest()
    print(f['path'], h == f['sha256'])

Citation

@dataset{aliasit_pii_aliasit_pii_dataset_v3,
  title   = {aliasit-pii-dataset-v3},
  version = {6.0.0},
  note    = {79 PII entity types, multilingual, Italian-first},
}

Derived from the sources listed above, under their respective licences. The attribution notices must travel with it.