BCCard/MoAI-Privacy-Filter

MoAI-Privacy-Filter

MoAI-Privacy-Filter is a Korean and English privacy-related entity detection model built by full fine-tuning `openai/privacy-filter`. It recognizes 29 entity types and emits 117 BIOES token classes. The training data emphasizes financial services and customer-service/VOC text while also covering identity, security, and infrastructure scenarios.

The model detects entity spans but does not decide how they should be masked or retained. Applications can apply their own handling policy to each predicted label. This distinction is especially important for PORT and ORGANIZATION, which are non-PII disambiguation labels included in the output taxonomy.

On held-out validation, strict micro F1 is 0.9824 for ko and 0.9708 for en. On an independently generated Golden Set, strict micro F1 is 0.9732 for ko and 0.9650 for en.

1. Model Summary

The model version and dataset version use independent version numbers. This model is v3 and was trained on dataset v1.

2. Label Taxonomy

Each entity label has B-, I-, E-, and S- boundary classes. Together with O, the model therefore has 4 x 29 + 1 = 117 output classes. O means that the model predicts no taxonomy entity at that token; it does not guarantee that the surrounding text is non-sensitive.

3. Usage

import torch
from transformers import AutoModelForTokenClassification, AutoTokenizer

model_id = "BCCard/MoAI-Privacy-Filter"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForTokenClassification.from_pretrained(model_id)
model.eval()

text = "고객 모아이님(900101-1234569)께서 010-0000-0000로 연락 요청하셨습니다."
encoded = tokenizer(
    text,
    return_offsets_mapping=True,
    add_special_tokens=False,
    truncation=True,
    max_length=1024,
    return_tensors="pt",
)
offsets = encoded.pop("offset_mapping")[0].tolist()

with torch.no_grad():
    logits = model(**encoded).logits.float()

print(tuple(logits.shape))  # (1, sequence_length, 117)

Apply constrained BIOES Viterbi decoding to logits[0], then map the decoded token spans to the original text with offsets. Character-span records can then be represented in the following form.

[
  {'start': 3, 'end': 6, 'label': 'PERSON'},
  {'start': 8, 'end': 22, 'label': 'RRN'},
  {'start': 26, 'end': 39, 'label': 'PHONE'}
]

Character offsets use Python's half-open interval [start, end). A downstream application could render those spans as follows, but this replacement behavior is not part of the model.

고객 [PERSON]님([RRN])께서 [PHONE]로 연락 요청하셨습니다.

3.1. Decoding

The reported metrics use constrained Viterbi decoding over the BIOES transition grammar, followed by whitespace boundary refinement. Independent per-token argmax can emit invalid BIOES sequences and is not the reported decoding path.

The bundled viterbi_calibration.json contains six transition biases. The default operating point sets all biases to zero, so BIOES transition constraints remain active without an additional precision-recall adjustment. Convert logits to FP32 before decoding.

Use a decoder that implements this BIOES constraint contract. The upstream `openai/privacy-filter` project provides the reference implementation and decoding behavior on which this model is based.

For batches, use right padding and pass only input_ids and attention_mask to the model. Keep offset_mapping outside the model for character-span reconstruction. Inputs longer than 1024 tokens were not represented in the training regime and should be chunked with enough overlap for the target use case.

4. Training

4.1. Data

The Golden Set was not used for training, checkpoint selection, or calibration. The training and validation data contain all 29 labels in both languages. English represents 25.74% of train and 26.03% of validation.

The dataset combines relabeled rows from `ai4privacy/pii-masking-openpii-1.5m` with Korean perturbation, English replay, and statically authored synthesis rows. Synthesis covers positive, confusion, hard-negative, weak-cue, long-context, and multi-label scenarios. The dataset is designed as synthetic training data and contains no operational customer records.

4.2. Procedure

<div align="center"> <img src="figures/evaluation-train-1-1.png" alt="Training loss, learning rate, and gradient norm by training step"> </div>

The selected FP32 checkpoint was exported as a deployment artifact in BF16. Artifact validation found 132 BF16 tensors and 8 FP32 attention sinks tensors.

4.3. Included Files

5. Evaluation

5.1. Setup

The validation split was used for checkpoint selection and experiment comparison. Final generalization was measured on a separately generated Golden Set containing weak-context entities, surface-similar decoys, label-confusion pairs, and boundary variants.

The headline metrics are language-slice strict micro Precision, Recall, and F1. A predicted entity is correct only when both its label and complete span boundary match the reference. Golden evaluation uses constrained Viterbi decoding and whitespace boundary refinement.

5.2. Results

<div align="center"> <img src="figures/evaluation-test-1-1.png" alt="Validation precision, recall, micro F1, and macro F1 by training step"> </div>

F1 Difference is Golden F1 minus Validation F1. The smaller Golden scores indicate a limited generalization decrease of 0.92 percentage points for Korean and 0.58 percentage points for English.

5.3. Error Characteristics

The aggregate results do not mean that every label performs equally. Error analysis of the English Golden slice shows the most visible weaknesses in ACCOUNT_NUMBER, ZIPCODE, PORT, and ORGANIZATION. Frequent confusion directions include PORT versus ZIPCODE, ACCOUNT_NUMBER versus BUSINESS_ID or IPIN, and ORGANIZATION versus PERSON.

These patterns are consistent with labels that share numeric shapes or require contextual role information. Downstream systems should evaluate per-label behavior on their own traffic, especially when label identity changes the handling action.

6. Intended Use

Suitable uses include:

• —Detecting privacy-related entities before Korean or English text is sent to an LLM or another downstream service.
• —Supporting offline privacy review of customer-service text, documents, email, and logs.
• —Producing typed entity spans for an application-specific masking, routing, retention, or review policy.

The model is not a complete anonymization system, a legal-compliance guarantee, or a substitute for domain-specific review. It should not be used as the sole control for high-impact decisions. Applications remain responsible for deciding whether each detected label is masked, transformed, retained, or escalated.

7. Limitations

• —Synthetic evaluation - Training, validation, and Golden data are synthetic. Performance on real customer text, OCR noise, slang, novel obfuscation, and unseen document structures has not been established.
• —Label-specific variation - High aggregate F1 can hide weaker labels and confusion pairs. ACCOUNT_NUMBER, ZIPCODE, PORT, and ORGANIZATION require particular attention based on the current Golden analysis.
• —Non-PII labels - PORT and ORGANIZATION are deliberately predicted even though they are not PII. Consumers must not assume that every non-O label requires the same action.
• —Context and boundary sensitivity - Weak contextual evidence, shared numeric formats, and long entity boundaries can produce missed entities, boundary errors, or label swaps.
• —Long inputs - The base architecture supports a larger context, but training examples were limited to 1024 tokens and the observed dataset maximum was 801 tokens. Longer inputs require separate validation and should normally be chunked.
• —Registry-backed identifiers - Synthetic account, telephone, passport, user, and social-security values cannot be exhaustively checked against private issuance registries. Any coincidental match with a real value is unintended.
• —Language and domain scope - Evaluation covers Korean and English with emphasis on financial, customer-service/VOC, identity, security, and infrastructure contexts. Other languages and domains are unsupported.

8. License, Attribution, and Citation

The model is released under the Apache 2.0 license. Its training dataset is released under CC BY 4.0 and is derived from ai4privacy/pii-masking-openpii-1.5m; follow the dataset card for its attribution requirements.

@misc{bccard2026moaiprivacyfilterv3,
  title        = {MoAI-Privacy-Filter v3: Korean and English Privacy-Related Entity Detection},
  author       = {BC Card},
  year         = {2026},
  howpublished = {https://huggingface.co/BCCard/MoAI-Privacy-Filter},
  note         = {Full fine-tune of openai/privacy-filter on BCCard/privacy-filter-openpii-masking v1}
}

Related resources:

• —Base model: `openai/privacy-filter`.
• —Training dataset: `BCCard/privacy-filter-openpii-masking`.
• —Upstream dataset: `ai4privacy/pii-masking-openpii-1.5m`.

9. Disclaimer

This model is provided as is, without warranties of accuracy, completeness, non-infringement, or fitness for a particular purpose. Users are responsible for testing the model in their own environment and ensuring that its use complies with applicable laws, regulations, contractual obligations, and organizational policies.