arthrod/gliner-opf-ptbr-pii-bench-v1

PT-BR PII Benchmark v1 — head-to-head

Why this exists Open-sourced at the request of @arthrod (Arthur Souza Rodrigues) after a two-night sprint training and benchmarking these models on an AMD MI300X. The motivation: there's surprisingly little published head-to-head data comparing MoE-based PII detectors (openai/privacy-filter) against dense small-model approaches (GLiNER on mmBERT/ettin) on a real-world Portuguese task — and the trade-offs turned out to be sharp enough to be genuinely useful for anyone building production redaction pipelines. The headline finding: a 30–60× false-positive rate gap between the recall-leaning mmBERT-small (×3) and the precision-leaning opf-fine-tune on the same negative-evidence corpora. mmBERT averages 2.4–4.6 false positives per spam email; opf averages 0.00–0.10. mmBERT wins partial F1 on PII content (avg 0.86 vs opf's 0.74), but if your downstream cost is high for false-positive redactions (legal docs, customer data flowing to LLMs, compliance scans you can't manually review), opf's precision pays off massively. Published openly so others can reproduce the comparison, extend it to other languages, or run their own model under the same protocol. Code + predictions + raw metrics all here.

Comprehensive cross-model benchmark for Brazilian-Portuguese PII detection, comparing four production models on the same val + holdout sets.

Models compared

What's in this repo

Top-level reports

• —stats_summary.md — comprehensive benchmark (footprint, throughput, latency dist, VRAM dist, quality, FP rates)
• —comparison_report.md — apples-to-apples val_5k comparison restricted to 24 PT-BR labels (nervaluate)
• —learning_curves.png — opf 9-checkpoint trajectory vs mmBERT 20-eval trajectory
• —WAKEUP_NOTE.md — overnight run notes from the opf training session

Machine-readable metrics

• —perf_metrics.json — full per-model performance (load, throughput, latency dist, VRAM dist)
• —quality_metrics.json — per-source F1 + per-source false-positive rates
• —hallucination_test_mmbert.json — mmBERT spurious-prediction test on 2,326 spam/phishing rows at threshold 0.4

Predictions (raw inference outputs)

• —predictions/holdout_5k/preds_<model>.jsonl — 4 models × 4,837 rows × 25 canonical labels at threshold 0.5
• —predictions/val_5k/preds_<model>.jsonl — 4 models on the 5k PT-BR val (also includes preds_base.jsonl for the un-finetuned opf checkpoint)

Code

• —scripts/benchmark_all_models.py — the comprehensive benchmark driver
• —scripts/test_hallucinations.py — spurious-prediction test on negative sources
• —scripts/compare_models_unified.py — val_5k unified F1 comparator
• —scripts/plot_opf_vs_mmbert_curves.py — learning curve plot
• —scripts/predict_25canonical_holdout5k.py — earlier (lighter) prediction script
• —scripts/run_full_bench.sh, run_hallucination_test.sh, etc. — ROCm sg-wrapper launchers

Headline result

mmBERT generates ~50–75 false positives per 100 spam-email rows; opf generates 0–8. mmBERT wins recall-driven F1 on PII sources by 0.06–0.20 partial-F1.

A note on the VRAM numbers

You'll see two columns related to GPU memory in stats_summary.md. They come from different torch.cuda.* APIs and answer different questions:

• —weights GB (max_memory_allocated()): peak live tensor memory — weights + activations actually in use
• —reserved GB (max_memory_reserved()): total allocator-pool size — what PyTorch's caching allocator holds onto for reuse and what the GPU actually charges you for

Both are sampled after warmup (one full forward pass on a representative input). The reserved column is the operationally honest "cost on the card" number; the allocated column is what the model needs at any moment in time.

For long-tail or high-throughput batched inference the gap closes; for one-shot serving the reserved column dominates.

Reproduction

All data, code, predictions, and metrics are here. To re-run:

git clone https://github.com/arthrod/gliner-training-utils  # or local working copy
cd gliner-training-utils
# install deps (.venv with torch + gliner + opf + nervaluate + matplotlib)
bash scripts/run_full_bench.sh

Hardware used: AMD MI300X single-GPU partition (ROCm 7.2). All models loaded in fp32 weights for fair comparison.

Eval protocol

• —Holdout: 11-source proportional 5,000-sample mixed corpus (4 PII + 7 spam/phishing)
• —Val (separate): 5,000-row PT-BR validation slice
• —Threshold: 0.5 (default), flat_ner=True, multi_label=False
• —Metrics: per-row latency timed with torch.cuda.synchronize() boundaries; VRAM via torch.cuda.max_memory_allocated/reserved reset per row
• —Quality: nervaluate partial+exact F1 on PII sources; FP-per-row + %-rows-with-≥1-FP on negative sources
• —Same val text + same 24 canonical PT-BR labels for all models (apples-to-apples)