labrat-aiko/popia-compliance-nli

POPIA Compliance NLI

Hand-authored Natural Language Inference pairs for South Africa's Protection of Personal Information Act (POPIA). Each example is a (premise, hypothesis, label) triple covering one of 7 canonical POPIA clauses. Designed for fine-tuning small cross-encoder NLI models into deterministic, local, auditable compliance judges — the alternative to prompting an LLM-as-judge over personal information.

Used to fine-tune `labrat-aiko/nli-popia-v1`, which reaches macro F1 0.813 on the pinned test split (vs 0.517 for the stock cross-encoder/nli-MiniLM2-L6-H768 base).

Splits

SHA-256 of popia_eval.jsonl: 120e14a55bb653f4e7ce49e10c3618aaef21a064b63ca831def92aa244935461

Clauses covered (all splits)

1. 1.POPIA consent — explicit, informed, opt-in consent per purpose
2. 2.POPIA minimality — collect only what's necessary for the stated purpose
3. 3.POPIA security safeguards — reasonable technical/organisational measures
4. 4.POPIA breach notification — notify the Information Regulator and data subjects without delay
5. 5.POPIA cross-border transfers — transfer only where lawful conditions are met
6. 6.POPIA general processing — lawful, fair, transparent processing tied to a legitimate purpose
7. 7.POPIA data subject rights — access, correction, deletion, objection

Each clause has train examples with entailment, contradiction, and neutral labels in roughly balanced proportions (see label counts below).

Label distribution (test)

Schema

Every row is a JSON object with:

{
  "clause": "POPIA consent",
  "premise": "Our signup flow presents a pre-ticked marketing opt-in checkbox...",
  "hypothesis": "The responsible party is relying on valid, freely-given consent for marketing.",
  "label": "contradiction",
  "scenario": "retail-loyalty"
}

• —clause: canonical clause name (one of 7)
• —premise: an LLM-shaped text — e.g. a customer-support reply, signup flow description, KYC confirmation, breach email. Distinct from raw statutes.
• —hypothesis: a plain-English restatement of what the clause requires, in progressive tense (NLI cross-encoders score progressive tense more reliably than imperatives — see model card).
• —label: one of entailment, contradiction, neutral — matches base-model config.id2label ordering {0: contradiction, 1: entailment, 2: neutral}.
• —scenario: short tag describing the domain the example is drawn from (e.g. retail-loyalty, news-fintech-onboarding, healthcare-breach).

Methodology — why hand-authored?

Scraped legal text is the wrong distribution. POPIA compliance in practice is judged on LLM-shaped text — support replies, KYC confirmations, breach emails — not on statutes. The model has to generalise from how-people-actually-talk-about-personal-data, not how-the-Act-is-worded.

Synthetic generation would poison the eval. If GPT-4 writes the training data and GPT-4 writes the outputs being validated in production, you measure GPT-4's self-consistency, not POPIA compliance.

180 pairs is enough for cross-encoder fine-tuning over 7 clauses. The base model already speaks English; you're teaching it a narrow decision boundary, not a new language.

Intended use

• —Fine-tune a cross-encoder NLI model for local, deterministic POPIA validation (see training recipe in the semantix-ai repo).
• —Benchmark existing NLI / LLM-as-judge systems on POPIA-shaped compliance tasks.
• —Build domain-specific extensions — your KYC flow, your dialect, your adversarial inputs — on top of these seeds.

Limitations

• —One author, one weekend. 180 training pairs is sufficient to move a cross-encoder meaningfully but is not a substitute for a corpus built from real customer production data.
• —English only. No Afrikaans, isiZulu, Sesotho, or other SA official languages — an obvious gap and a priority for v2.
• —Test set is hand-authored by the same author as train. There may be stylistic correlation even across separate authoring sessions. External validation welcome.
• —Not a substitute for a DPIA, a POPIA specialist, or the Information Regulator's guidance. This is a validation primitive, not a compliance replacement.

Citation

@misc{popia-compliance-nli-2026,
  author       = {Eland, Akhona},
  title        = {POPIA Compliance NLI: Hand-authored NLI pairs for POPIA-specific validation},
  year         = 2026,
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/labrat-aiko/popia-compliance-nli}},
  note         = {Apache 2.0}
}

License

Apache 2.0. Commercial and derivative use is explicitly permitted. If you publish a fine-tuned sibling (e.g. GDPR, HIPAA), a link back is appreciated but not required.

Related

• —Fine-tuned model: `labrat-aiko/nli-popia-v1`
• —Python library: `semantix-ai` — ships POPIAJudge and 7 pre-configured clause Intent instances.
• —Article: I Fine-Tuned a Compliance Judge and Beat the Stock Model by +29.6pp F1