Datasetpublic

kierandesmond/spanish-gdpr-pii-ner-v9-validation

Spanish GDPR PII NER — v9 Validation & Reproducibility Held-out evaluation set, scripts, and results for kierandesmond/spanish-gdpr-pii-ner-v9. Why this repo exists v6/v8 were scored on a tiny n=1–6 "battery" that reported 90%+ but hid real failures. v9 introduces a large held-out eval set (7,304 examples, 200–336 per entity, with hard negatives) to get statistically-meaningful per-entity numbers. Re-scoring v8 on it revealed SIP_CARD 0.5%, ETHNIC_ORIGIN 40%… See the full description on the dataset page: https://huggingface.co/datasets/kierandesmond/spanish-gdpr-pii-ner-v9-validation.

sourceHugging Facecc-by-4.0updated 4mo agoView on Hugging Face

0likes146downloads

Dataset Card

Spanish GDPR PII NER — v9 Validation & Reproducibility

Held-out evaluation set, scripts, and results for `kierandesmond/spanish-gdpr-pii-ner-v9`.

Why this repo exists

v6/v8 were scored on a tiny n=1–6 "battery" that reported 90%+ but hid real failures. v9 introduces a large held-out eval set (7,304 examples, 200–336 per entity, with hard negatives) to get statistically-meaningful per-entity numbers. Re-scoring v8 on it revealed SIPCARD 0.5%, ETHNICORIGIN 40%, GENETIC_DATA 70%, etc.

Headline results (span-level pass-rate)

Metric	v8	v9
Macro positive pass-rate (28 entities)	87.3%	99.55%
Macro negative pass-rate	76.6%	100%
Lowest single entity	0.5%	94.6%

All 28 entities ≥ 90% in v9. See comparison_v8_v9.csv for the full table.

Files

File	Description
`eval_v9.jsonl`	The held-out eval set (tokens + BIO tags + entity + pos/neg kind + source)
`build_eval.py`	Builds the eval set (real corpora held-out splits + eval-only synthetic families)
`train_v9.jsonl`	v9 training data (13,086 examples)
`build_train.py`	Builds training data (targets weak entities + hard negatives)
`special_phrasings.py`	Register-varied templates; `split_families()` enforces train/eval disjointness
`train_v9.py`	Training script (continues from v8, Trackio, 57-label schema asserted)
`score.py`	Span-level scorer (the headline metric)
`eval_scores_v8.{csv,json}`	Honest v8 baseline
`eval_scores_v9.{csv,json}`	v9 results
`comparison_v8_v9.csv`	Per-entity v8→v9 delta
`confusion_v9.csv`, `confusion_matrix_v9.png`	Sensitive-entity confusion (0 SEXUAL↔RELIGIOUS, 0 POLITICAL↔UNION)

Reproduce

bash

pip install transformers==4.57.6 datasets accelerate seqeval scikit-learn pandas pyarrow
python build_eval.py          # -> eval_v9.jsonl
EVAL_MODEL=kierandesmond/spanish-gdpr-pii-ner-v9 EVAL_TAG=v9 python score.py

Anti-leakage discipline

—Synthetic special-category: eval uses template families disjoint from training (split_families).
—Real corpora: eval uses validation/test splits; training uses train splits.
—Verified: 0 exact eval-family sentences leaked into training.

Caveats

Special-category entities have no labeled Spanish corpus and are synthetic-trained. Eval measures generalization to unseen carrier sentences but cannot fully represent production free-text. Treat special-category detection as assistive (flag-for-review) until validated on real domain documents.