neondijital/neonredact-tr-bench

NeonRedact-TR Bench v1

An independent evaluation set for personally identifiable information (PII) detection in Turkish text.

400 fictional Turkish documents, 1,617 gold entities, 24 labels. Court records, bank receipts, insurance policies, e-Devlet printouts, hospital reports, payslips, WhatsApp chats, server logs.

Results for NeonRedact, OpenAI Privacy Filter and OpenMed v2 are in [`RESULTS.md`](RESULTS.md).

The split is stratified by document category with a fixed seed (20260924). File hashes are in SHA256.json. v1 will not change; corrections will ship as v1.1.

Why this exists

Turkish attaches case suffixes to names with an apostrophe (Ayşe Öztürk'ün, Konak'tan), writes addresses in its own order, and uses identifiers that do not exist in EU or US datasets: T.C. national ID, tax number (VKN), Turkish IBAN, licence plates. Open PII models are mostly evaluated on English. This set measures what they do on Turkish.

Format

{"id": "005", "kategori": "log", "tur": "log", "text": "2026-09-18 14:22:07 | KAYIT 1 | Isim: Tuncay Er | TCKN: 2860 ...", "entities": [{"start": 38, "end": 47, "label": "KISI", "text": "Tuncay Er", "puanli": true}]}

start and end are character offsets. puanli marks the 17 scored labels.

Labels

Scored (17): KISI person, ADRES address, TELEFON phone, EPOSTA email, DOGUM_TARIHI date of birth, OLAY_TARIHI personal event date, TC_KIMLIK national ID, VKN tax ID, IBAN, KART card, PLAKA plate, KURUM organization, MESLEK occupation, AKRABALIK kinship, SAGLIK health, TARAF_SIFATI legal role, GIZLI secret.

Not scored (7): SICIL_NO, RUHSAT_NO, SASI_NO, MOTOR_NO, EHLIYET_NO, BELGE_NO, KULLANICI_ADI. Identifiers common in Turkish documents that no evaluated model covers.

How it was built

• —The NeonRedact training data generator and its templates were not used.
• —Document structures were taken from real Turkish documents. No real values were used; every name, address and number is fictional.
• —Drafts were written with a language model; every entity was checked by hand.
• —Automatic checks, all passing with zero errors: T.C. ID and tax ID checksums, IBAN mod-97, Luhn for cards, plate format (province code 01 to 81, letter/digit length rules, forbidden letters), suffix never inside a span, titles never inside a person span, no number reused for two different people, no overlapping spans.

Labelling rules (summary)

1. 1.Case suffixes are outside the span, including on numbers and dates (0544 802 61 39'u).
2. 2.Titles (Dr., Av., Bey, Hanım) are outside the person span.
3. 3.An address is labelled only when it is tied to a person. Branch offices, venues, accident locations are not.
4. 4.Shared corporate contacts (info@, call centres) are not labelled. A person's work email is.
5. 5.IBAN and tax ID are always labelled, even for companies: sole proprietorships cannot be told apart from text.
6. 6.Masked numbers (**** 4417) are not labelled.
7. 7.One-time codes (SMS OTP, 3D Secure, reset links) are not labelled. Persistent secrets (passwords, CVV, API keys, recovery codes) are GIZLI.
8. 8.Commercial dates (order, invoice, delivery) are not labelled.
9. 9.Health data must be specific: a condition, diagnosis, treatment or procedure.
10. 10.Public and historical figures are not labelled.
11. 11.Ambiguous examples were rewritten or removed.

Full rules in Turkish: `ETIKETLEME_KURALLARI_TR.md`.

Traps

About 20% of documents contain no PII at all. Others embed hard negatives: a cargo number that passes the T.C. ID checksum, order and campaign codes that pass Luhn, the public test card 4111 1111 1111 1111, example IBANs, streets and bridges named after people, and Turkish names that are also common words (Deniz, Gül, Umut). Several sensitive concepts appear twice, once tied to a person and once as general information.

Metrics

Per-label and micro F1 (exact and partial overlap), 95% bootstrap confidence intervals, false-alarm rate on PII-free documents, per-category F1, and suffix leakage: among suffixed entities a system finds, how often it includes the suffix in the span.

Code: eval_tahmin.py (runs systems), eval_puan.py (scores).

Known limitations

• —Written by the author of NeonRedact. Independent of its training data, but a home-field advantage cannot be ruled out.
• —Medication names are not labelled, although a drug name can reveal a diagnosis.
• —Age and postal code are not in the schema.
• —GIZLI (22), TARAF_SIFATI (29) and VKN (29) have fewer than 30 test examples.
• —Text is fictional and does not reproduce real-world document frequencies.

Citation

@misc{neonredact_tr_bench_2026,
  title  = {NeonRedact-TR Bench: An Evaluation Set for Turkish PII Detection},
  author = {Kömürcü, Aykan},
  year   = {2026},
  url    = {https://huggingface.co/datasets/neondijital/neonredact-tr-bench}
}

Built by NeonDijital.