bengid/pii-redaction-deberta-base

DeBERTa-v3-Base PII Redaction

Fine-tuned microsoft/deberta-v3-base for Named Entity Recognition targeting 27 PII entity types. Trained on the English subset of ai4privacy/pii-masking-300k with a class-weighted CrossEntropyLoss. Achieves 0.9557 macro-F1 on the validation set.

Recommended when accuracy is the priority and compute is not a constraint. Best overall performance (macro F1: 0.9557), particularly on rare entity types. Well-suited for offline batch processing, compliance pipelines, or any server-side deployment where an extra ~5ms of latency is acceptable.

Usage

from transformers import pipeline

pipe = pipeline(
    "token-classification",
    model="bengid/pii-redaction-deberta-base",
    aggregation_strategy="first",
    device=0  # omit for CPU
)

text = "She lives at 742 Evergreen Terrace, Springfield, IL 62704."
entities = pipe(text)
print(entities)

Training Data

Filtered subset of ai4privacy/pii-masking-300k, restricted to English-language examples only (language == "en"). The full dataset is multilingual; this model targets English text only.

Preprocessing:

• —Dropped CARDISSUER entity class (little support)
• —Validation set split 50/50 into validation and test

Training Procedure

Two-phase Fine-tuning (frozen backbone → unfrozen) from `microsoft/deberta-v3-base` using a weighted token-classification trainer and stage-specific learning rates.

Hyperparameters

Evaluation

Evaluated on the English validation subset (3,973 examples) at the best checkpoint.

Per-Entity F1

Limitations

• —English only — trained exclusively on English text; performance on other languages is undefined.
• —Max 512 tokens — inherited from DeBERTa's positional embeddings. Longer documents should be chunked.
• —Name entities are harder — The model underperforms on GIVENNAME and LASTNAME entities:

Likely causes: performance correlates strongly with training support — LASTNAME1/GIVENNAME1 (primary occurrences, ~900-1100 examples) score significantly higher than LASTNAME2/3 (secondary/tertiary occurrences, 105-313 examples). Additionally, names are inherently context-dependent: without surrounding cues like titles or formal structure, the model has less signal to distinguish them from non-PII tokens — even the best-supported name entities (LASTNAME1, GIVENNAME1) fall notably below the macro F1 of 0.9557, suggesting names are a structurally harder category regardless of support.

• —Not a redaction tool by itself — this model detects and labels PII spans; downstream redaction/masking logic must be implemented separately.
• —Subword labeling convention — following the HuggingFace token classification convention, only the first subword of each word was assigned its NER label during training; continuation subwords were assigned -100 (ignored by the loss). The practical consequence is that the model predicts O with high confidence on continuation subwords, which can cause partial detection of multi-subword entities (e.g. john@example.com returned as only john) when using aggregation_strategy="simple". Use aggregation_strategy="first" for inference, which is consistent with this training convention.

Intended Use

Intended uses:

• —Detecting and labeling PII spans in English text for downstream redaction or pseudonymization pipelines.
• —Privacy compliance tooling (GDPR, CCPA, HIPAA).
• —Pre-processing step before storing or sharing user-generated content.

Out-of-scope uses:

• —Non-English text.
• —Real-time high-stakes medical or legal decision-making without human review.
• —As a sole compliance mechanism — model errors are expected; human auditing is recommended.

Model Comparison

[1] see DeBERTa-v3-XSmall PII Redaction for latency explanation

License

The model weights are released for research and non-commercial use, consistent with the training data license (ai4privacy/pii-masking-300k). Users should review the dataset license before commercial deployment.

Citation

If you use this model, please cite the base model architecture and the training dataset:

Base model (DeBERTa-v3):

@misc{he2021debertav3,
      title={DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing}, 
      author={Pengcheng He and Jianfeng Gao and Weizhu Chen},
      year={2021},
      eprint={2111.09543},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Training dataset:

@misc{ai4privacy2023pii,
  title     = {PII Masking 300k},
  author    = {Ai4Privacy},
  year      = {2023},
  publisher = {Hugging Face},
  doi       = {10.57967/hf/1995},
  url       = {https://huggingface.co/datasets/ai4privacy/pii-masking-300k}
}