ScienceSoft/scnsoft-pii-encoder-small

ScienceSoft PII encoder — small

A fast, multilingual token-classification model for personal data without a fixed format: names of people and organisations, locations, addresses and dates. It is a 6-layer, 384-wide student of `ScienceSoft/scnsoft-pii-encoder`, and it runs 3.9× faster than that model on CPU.

It is considerably less accurate than the model it was distilled from. Its recall matches the full model's, but its precision does not: on real text it marks far more words that are not personal data. Before training, a bar was set for releasing a small model: an average PII Bench F1 of at least 0.72, with no language more than 0.08 below the full model. This model does not meet that bar. It is published with that result stated, so the trade between speed and accuracy can be judged on the numbers.

Architecture: `microsoft/Multilingual-MiniLM-L12-H384`, reduced to 6 of its 12 layers (layers 1, 3, 5, 7, 9 and 11), with XLM-R's tokenizer. It is trained on the full model's synthetic corpus with the full model as teacher, then exported to ONNX and quantised to int8. The file is 107 MB, and inference runs on CPU.

Labels

PERSON ORGANIZATION LOCATION ADDRESS DATE_TIME (BIO tagging, 11 classes), the same set and order as the full model.

Identifiers with a fixed format (email addresses, telephone, payment-card, IBAN and passport numbers) are not labels; a regular expression recognises them exactly. The full model is trained to leave them unmarked. This model does so much less reliably; see touch rate below.

Speed

Measured on 300 PII Bench rows:

• —int8 ONNX Runtime 1.19.2 on CPU, batch size 1;
• —all three models interleaved row by row;
• —an Apple M4 Max with 16 cores, a shared workstation with a 1-minute load average of 18–19.

Absolute milliseconds depend on the host; the ratio is the informative part.

Evaluation on PII Bench

PII Bench (piibench) scores detection of the five entity types above in text serialised the way outbound traffic carries it (JSON, XML, SQL, CSV, key=value, log lines, Markdown tables, OCR output) as well as prose.

• —It comprises 3,500 rows, 500 from each of seven third-party datasets.
• —Precision, recall and F1 are computed over characters and are label-agnostic.
• —Scores are micro-averaged within each dataset and then averaged across the seven.
• —Leaderboard: ScienceSoft/piibench-leaderboard.

How these numbers were produced. All models in the tables below are scored the same way: by the production Rust inference engine, before its post-filters. That is the same path as the leaderboard's scnsoft-pii-encoder-engine-raw row, which this scoring reproduces exactly (0.7725). The intervals are 95 % percentile bootstrap intervals over rows, resampled within each dataset (1,000 resamples).

Conflict of interest. PII Bench is built and published by ScienceSoft, the developer of this model. Four of its seven datasets (TAB, FactRuEval, MEDDOCAN, IAHLT) informed changes to the training corpus this model shares with the full model.

Where the accuracy is lost.

• —Recall is the full model's (0.902 against 0.908). The loss is in precision (0.458 against 0.685), and it appears in every dataset and in every free-text format.
• —On an in-distribution synthetic evaluation, drawn from the training generator, the model scores F1 0.986, close to the full model's 0.993. It learned the task on the kind of text it was trained on. It fires too often on real text, which its training data (synthetic, like the full model's) never showed it.
• —Distillation did not close the gap, because the teacher's guidance was also only given on synthetic text.
• —Touch rate is the share of characters belonging to format-fixed identifiers that a model marks anyway: 0.237 here, against 0.020 for the full model. In a product that already masks those identifiers by regular expression, this costs little. As a stand-alone detector, it over-marks them.

Variability across training seeds

Two training runs of the same recipe, differing only in the random seed:

The two runs agree to within 0.007 F1. Seed 1234 is published on main by convention, not because it scored better; it did not.

Training

• —Corpus: the same synthetic corpus as scnsoft-pii-encoder (v23), rebuilt from that run's own generator snapshot and configuration and verified row for row by count: 155,141 training and 5,300 evaluation rows.
• —Distillation from `scnsoft-pii-encoder`: the loss is 0.5 × cross-entropy on the gold labels plus 0.5 × T² × KL(teacher ‖ student) on the teacher's token distributions at temperature T = 2, over non-padding tokens.
• —Schedule: two epochs, 19 minutes on one GPU.
• —Stack: transformers==4.57.1, tokenizers==0.22.1, torch==2.9.0a0, faker==37.12.0.
• —int8 against fp32: −0.0037 F1 on the synthetic evaluation, 95 % CI [−0.0055, −0.0022].

Limitations

• —Below the release bar in every language (see above). Where accuracy matters more than latency, use `scnsoft-pii-encoder`.
• —Low precision on real text. Expect far more false positives than from the full model, and many more format-fixed identifiers marked.
• —Trained only on synthetic data, like the full model. Scores on generated text overstate real-world ability.
• —The limitations of the full model apply too. Arabic is its weakest language; Arabizi is a reconstruction; DATE_TIME is kept by the product only next to a birth-date cue.

Use

Required files: config.json, tokenizer.json, onnx/model_quantized.onnx.

import json, onnxruntime as ort
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("ScienceSoft/scnsoft-pii-encoder-small")
cfg = json.load(open("config.json"))
id2label = {int(k): v for k, v in cfg["id2label"].items()}
sess = ort.InferenceSession("onnx/model_quantized.onnx",
                            providers=["CPUExecutionProvider"])

enc = tok(text, return_offsets_mapping=True, return_tensors="np",
          truncation=True, max_length=256)
offsets = enc.pop("offset_mapping")[0]
logits = sess.run(None, {i.name: enc[i.name] for i in sess.get_inputs()})[0]
# argmax -> id2label -> merge BIO spans using `offsets` for character positions

The ONNX graph takes input_ids and attention_mask only. Encode with the special tokens <s> and </s>, as the model was trained; AutoTokenizer.__call__ adds them by default.

Provenance

The training corpus is synthetic text produced by ScienceSoft's own generator; no production traffic and no scraped text is used for training. Entity values come from project-written lists and from Faker (person, company and city names), libpostal (address vocabulary) and GeoNames (Arabic-script city names).

Licence

Proprietary to ScienceSoft. The weights are published so that the results on this page can be evaluated and reproduced; no licence to use them in production, to modify them or to redistribute them is granted. For licensing, contact ScienceSoft (https://www.scnsoft.com).

Basis for these terms.

• —Initialised from microsoft/Multilingual-MiniLM-L12-H384 (MIT licence).
• —Uses the tokenizer of FacebookAI/xlm-roberta-base (MIT licence).
• —Distilled from ScienceSoft's own scnsoft-pii-encoder.
• —Training corpus includes entity values from Faker (MIT), libpostal (MIT) and GeoNames (CC BY 4.0).

All of these licences permit a derivative work to be distributed under proprietary terms, provided their notices are retained and GeoNames is attributed. None is share-alike, and none restricts commercial use. The notices and the attribution are in THIRD_PARTY_NOTICES.md.