felipe53/slm-deid-name-judgment

slm-deid — context-sensitive personal-name judgment (Qwen3-1.7B QLoRA)

A 4-bit QLoRA adapter over unsloth/Qwen3-1.7B-unsloth-bnb-4bit fine-tuned for one narrow, safety-critical behavior: deciding whether a name in educational text refers to a real person, and tagging only those spans. Given a short passage it returns the text byte-identical except that person-name spans are wrapped in ⟨NAME⟩…⟨/NAME⟩ — and nothing else is tagged.

This is the judgment core of de-identification — the part regex and constrained decoding structurally cannot do. Pattern PII (email / phone / SSN / IDs / URLs) and output format are out of scope here; they are owned by the deterministic pipeline that surrounds this model.

• —Canonical run: sft-v3-gpt551 — trained on data from a live teacher + independent verifier (not in-session templates). This is the credible reference line.
• —Repo: https://github.com/f15cubing/slm-deid — full eval harness, dataset, and methodology.

The behavior, by example

Same token, opposite call, decided by context. Run the side-by-side yourself with python -m src.demo --adapter <this-adapter>.

Base vs. tuned — the headline result

Evaluated on 51 quarantined hard cases (ambiguous names in context). Base = the prompted 4-bit Qwen3-1.7B on the same backend; tuned = base + this adapter. The comparison is prompt vs. fine-tune, which is the assignment's baseline. Full tables + 95% bootstrap CIs: docs/results.md → gpt551.

Per-category: person_vs_eponym recall 0.33 → 1.00, person_vs_place recall → 1.00, negative_trap pass 0.40 → 1.00. Generalization to names never seen in training was verified on a held-out-names probe (base→tuned recall 0.08 → 1.00) and an OOD probe (recall 0.05 → 0.89) — the judgment generalized, it did not memorize.

For context, a frontier API model (gpt-4.1) scored pass 0.88 on the same 51 hard cases; the 1.7B tunes are competitive with it and actually beat it on recall (the frontier model is precision-first and under-recalls). See docs/eval-engine-comparison.md.

Training setup

Config was frozen deliberately: a project rule forbids touching hyperparameters to paper over data problems — failure modes were fixed by generating targeted data, not by tuning lr/r/epochs.

Training data

818 train / 90 val (sft-v3-gpt551 splits). Generated by a live OpenAI-compatible teacher (via the TrueFoundry LLM Gateway) writing passages, plus an independent verifier pass that rejects teacher/verifier disagreements. Built from matched minimal pairs (person_vs_common / _place / _eponym / possessive) so the model sees the same token used both ways. Dataset card: docs/dataset-card-v3.md.

Quality-gate + leakage drops: 134 verifier-disagreement, 98 eval-surface-leak, 48 negative-trap-has-name, plus a few label-integrity rejects.

Eval-leakage — a hard ceiling, verified clean

The quarantined evaluation sets are physically separate and were never fed to the teacher, augmentation, or training. Independently re-verified post-hoc: 0 exact and 0 substring overlaps between the 818/90 training splits and all 201 quarantined eval inputs (hardcases / adversarial / held-out-names / OOD). Enforced in CI by tests/test_no_eval_leakage.py.

Intended use

• —De-identifying personal names in short English educational text (essays, chat, dialogue, narrative, exposition), as the judgment stage of a larger de-id pipeline.
• —Research / educational demonstration of reliability engineered through data vs. capability accessed through prompting on a narrow behavior.

Limitations & out-of-scope

1. 1.Names only. No address / DOB / MRN / other PII categories (those are pipeline-side).
2. 2.Byte-identity on messy real text. On clean text integrity is ~0.98; on very messy real prose (collapsed whitespace, zero-width chars) the "regenerate the passage verbatim" output format can drift — the surrounding pipeline's tag-by-offset projection is the fix (unwrap(project(...)) == original by construction). Prefer the projected path in production.
3. 3.Possessive contrast is the residual weak spot (over-tag/integrity on possessive, n=3).
4. 4.Small eval (n=51), single seed, single teacher+verifier. Per-category cells are noisy.
5. 5.English + educational register only. Not validated on clinical/legal text or other languages.
6. 6.Not a certified de-identification system; do not deploy on regulated data without human review.

How to use

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "Qwen/Qwen3-1.7B"  # or the 4-bit unsloth base on CUDA
tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE)
model = PeftModel.from_pretrained(model, "<this-repo-id>")

# Easiest path — the repo's Tagger contract (handles the non-thinking chat template):
#   from src.infer import load_hf_tagger
#   tagger = load_hf_tagger(adapter="<adapter-path>")
#   tagger.tag("Chelsea helped me revise my thesis.")
#   → "⟨NAME⟩Chelsea⟨/NAME⟩ helped me revise my thesis."

The model expects the system prompt in src/common/prompts.py and Qwen3's non-thinking chat template. See src/infer.py and src/demo.py in the repo.

Provenance & reproduction

• —Adapter: sft-v3-gpt551 (r=32/α=32, checkpoint-156).
• —Reproduce on Colab: notebooks/v3_colab_train_eval.ipynb — live teacher via --provider openai with OPENAI_BASE_URL/TEACHER_MODEL → frozen configs/train.yaml QLoRA → base-vs-tuned on eval/hardcases.
• —Source model cards (per run): docs/model-card-gpt551.md, docs/model-card-v3.md.
• —License: Apache-2.0 (inherits the Qwen3-1.7B base license).