ScienceSoft/scnsoft-pii-encoder-pruned-6l

ScienceSoft PII encoder — pruned to 6 layers

`ScienceSoft/scnsoft-pii-encoder` with its top six transformer layers removed and the remaining six retrained. The full model is used as teacher. It detects personal data without a fixed format (names of people and organisations, locations, addresses and dates) in seven languages, and it runs 1.8× faster than the full model on CPU.

It is less accurate than the full model. Its recall matches the full model's, but its precision does not: on real text it marks more words that are not personal data. Before training, a bar was set for releasing a faster model:

• —average PII Bench F1 of at least 0.72;
• —no language more than 0.08 below the full model;
• —at least 2× the full model's speed.

This model meets none of the three. It reaches 0.689 average F1 and 1.8× the speed. It is published with those results stated, so the trade between speed and accuracy can be judged on the numbers.

Architecture: xlm-roberta-base with 6 layers (the full model's layers 0–5), 768 wide, same tokenizer, labels and classifier head as the full model. Exported to ONNX and quantised to int8; the file is 235 MB (most of it the 250,002-token embedding table), and inference runs on CPU.

Labels

PERSON ORGANIZATION LOCATION ADDRESS DATE_TIME (BIO tagging, 11 classes), the same set and order as the full model. Identifiers with a fixed format (email addresses, telephone, payment-card, IBAN and passport numbers) are not labels. See touch rate below for how often this model marks them anyway.

Speed

Measured on 300 PII Bench rows:

• —int8 ONNX Runtime 1.19.2 on CPU, batch size 1;
• —all three models interleaved row by row;
• —an Apple M4 Max with 16 cores, a shared workstation with a 1-minute load average of 18–19.

Absolute milliseconds depend on the host; the ratio is the informative part.

Halving the depth roughly halves the compute per token. The width stays at 768, which is why the 6 × 384 model is faster still.

Evaluation on PII Bench

PII Bench (piibench) scores detection of the five entity types above in text serialised the way outbound traffic carries it (JSON, XML, SQL, CSV, key=value, log lines, Markdown tables, OCR output) as well as prose.

• —It comprises 3,500 rows, 500 from each of seven third-party datasets.
• —Precision, recall and F1 are computed over characters and are label-agnostic.
• —Scores are micro-averaged within each dataset and then averaged across the seven.
• —Leaderboard: ScienceSoft/piibench-leaderboard.

How these numbers were produced. All models below are scored the same way: by the production Rust inference engine, before its post-filters. That is the same path as the leaderboard's scnsoft-pii-encoder-engine-raw row, which this scoring reproduces exactly (0.7725). The intervals are 95 % percentile bootstrap intervals over rows, resampled within each dataset (1,000 resamples).

Conflict of interest. PII Bench is built and published by ScienceSoft, the developer of this model. Four of its seven datasets (TAB, FactRuEval, MEDDOCAN, IAHLT) informed changes to the training corpus this model shares with the full model.

Where the accuracy is lost.

• —Recall is the full model's (0.906 against 0.908). The loss is in precision (0.573 against 0.685), across every dataset (−0.05 to −0.12 F1) and every free-text format.
• —On an in-distribution synthetic evaluation, drawn from the training generator, the model scores F1 0.982, close to the full model's 0.993. It fires too often on real text, which its training data never showed it.
• —Touch rate, the share of characters belonging to format-fixed identifiers that a model marks anyway, is 0.160 here, against 0.020 for the full model.

Variability across training seeds

Two training runs of the same recipe, differing only in the random seed:

Seed 1234 is published on main by convention, not selected by score.

Training

• —Initialisation: the full model's embeddings, transformer layers 0–5 and classifier head. Keeping the lower layers follows the finding that dropping the top layers of a pretrained transformer holds up best after retraining (Sajjad et al., 2020, On the Effect of Dropping Layers of Pre-trained Transformer Models).
• —Corpus: the same synthetic corpus as scnsoft-pii-encoder (v23), rebuilt from that run's own generator snapshot and configuration and verified by count: 155,141 training and 5,300 evaluation rows.
• —Distillation from `scnsoft-pii-encoder`: 0.5 × cross-entropy on the gold labels plus 0.5 × T² × KL(teacher ‖ student) at T = 2, over non-padding tokens.
• —Schedule: two epochs, 31 minutes on one GPU.
• —Stack: transformers==4.57.1, tokenizers==0.22.1, torch==2.9.0a0, faker==37.12.0.
• —int8 against fp32: +0.0036 F1 on the synthetic evaluation, 95 % CI [+0.0020, +0.0052]. The int8 file scored slightly higher here.

Limitations

• —Below the release bar on average F1, on German, and on speed (see above). Where accuracy matters, use `scnsoft-pii-encoder`. Where speed matters most, `scnsoft-pii-encoder-small` is 2× faster again, at lower accuracy.
• —Lower precision on real text than the full model, and more format-fixed identifiers marked.
• —Trained only on synthetic data, like the full model. Its other limitations apply too: Arabic is the weakest language; Arabizi is a reconstruction; DATE_TIME is kept by the product only next to a birth-date cue.

Use

Required files: config.json, tokenizer.json, onnx/model_quantized.onnx.

import json, onnxruntime as ort
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("ScienceSoft/scnsoft-pii-encoder-pruned-6l")
cfg = json.load(open("config.json"))
id2label = {int(k): v for k, v in cfg["id2label"].items()}
sess = ort.InferenceSession("onnx/model_quantized.onnx",
                            providers=["CPUExecutionProvider"])

enc = tok(text, return_offsets_mapping=True, return_tensors="np",
          truncation=True, max_length=256)
offsets = enc.pop("offset_mapping")[0]
logits = sess.run(None, {i.name: enc[i.name] for i in sess.get_inputs()})[0]
# argmax -> id2label -> merge BIO spans using `offsets` for character positions

Encode with the special tokens <s> and </s>, as the model was trained; AutoTokenizer.__call__ adds them by default.

Provenance

The training corpus is synthetic text produced by ScienceSoft's own generator; no production traffic and no scraped text is used for training. Entity values come from project-written lists and from Faker (person, company and city names), libpostal (address vocabulary) and GeoNames (Arabic-script city names).

Licence

Proprietary to ScienceSoft. The weights are published so that the results on this page can be evaluated and reproduced; no licence to use them in production, to modify them or to redistribute them is granted. For licensing, contact ScienceSoft (https://www.scnsoft.com).

Basis for these terms.

• —Derived from ScienceSoft's own scnsoft-pii-encoder, which is derived from `FacebookAI/xlm-roberta-base` (MIT licence).
• —Training corpus includes entity values from Faker (MIT), libpostal (MIT) and GeoNames (CC BY 4.0).

All of these licences permit a derivative work to be distributed under proprietary terms, provided their notices are retained and GeoNames is attributed. None is share-alike, and none restricts commercial use. The notices and the attribution are in THIRD_PARTY_NOTICES.md.