LocalAI-io/privacy-filter-GGUF

privacy-filter — GGUF (F16 + Q8_0)

GGUF conversion of `openai/privacy-filter`, OpenAI's bidirectional PII token-classification model. It labels every token with a BIOES tag over 8 PII categories (33 classes) in a single forward pass, then decodes coherent spans with a constrained Viterbi procedure — so it can be served locally with no Python as the encoder/NER tier of a PII redactor.

For the full model description, training, evaluation, operating points, limitations, and citations, see the [source model card](https://huggingface.co/openai/privacy-filter) — this card only covers the GGUF packaging and how to run it.

For broader language coverage (54 categories across 16 languages), see the multilingual fine-tune `privacy-filter-multilingual` GGUF.

Runtimes

This GGUF uses a custom architecture, `openai-privacy-filter`, that is not (yet) part of upstream llama.cpp. It runs on:

1. 1.[privacy-filter.cpp](https://github.com/localai-org/privacy-filter.cpp) (recommended) — a small standalone GGML engine for exactly this model family, on stock upstream ggml with no patches (CPU / CUDA / Vulkan). This is the reference runtime.

   # build (see the repo README for CUDA/Vulkan)
   cmake --preset release && cmake --build --preset release -j
   # run
   echo "My name is Alice Smith" | \
     build/release/pf-cli --classify privacy-filter-f16.gguf 0.5

It exposes a flat C API (pf_load / pf_classify → entity spans with UTF-8 byte offsets; pf_tokenize / pf_logits) shaped for FFI — see the repo README.

1. 1.[LocalAI](https://github.com/mudler/LocalAI) — install from the model gallery; LocalAI serves it behind the gRPC TokenClassify RPC and runs the constrained BIOES Viterbi decode, returning entity spans. LocalAI drives it through the `privacy-filter` backend (which wraps privacy-filter.cpp). The model is not a chat/completion model — it is a PII detector that other models opt into via a pii.detectors list.

1. 1.llama.cpp — only with a patch. Stock llama.cpp, llama-cpp-python, Ollama, and LM Studio will fail to load this file (unknown model architecture: 'openai-privacy-filter'). The arch can be added with carry-patches (TOKENCLS pooling, the architecture + HF→GGUF converter, the bidirectional banded-attention graph, and an all-SWA no-cache mask fix; TOKENCLS pooling tracks the still-open PR #19725). Until that support lands upstream, privacy-filter.cpp above is the patch-free alternative.

Pooling note (llama.cpp path only): the model must be loaded with TOKEN_CLS pooling (the GGUF's default). If you drive llama-embedding directly for testing, do not pass --pooling none. privacy-filter.cpp handles this automatically.

Files

sha256 (f16): eb71312b6b9370d0fe582e576b840567bb06603c4de241c6d899205d1b04dc81 sha256 (q8): 80efc1803eda7c095a79741d2008c07e2e0a57b01bac8825fbeb448fd097998c

Q8_0 quantization — and why it isn't free. q8 stores the bulk of the weights (the MoE expert matrices) as 8-bit integers instead of 16-bit floats — via `scripts/requant_q8.py`, with attention, embeddings and the classifier head left at F16. That roughly halves the download (2.82 GB → ≈1.6 GB) and is usually a bit faster on CPU.

The catch: reducing precision throws information away, and it is almost never a free lunch. On a mixed-PII document (1,360 tokens) q8 matched f16 on 99.7% of token labels (average prediction shift, KL divergence, of 1.1e-3) — close, but note it did not match on all of them; a few tokens flipped. That is the point in miniature: a reassuring average still hides the specific cases that change, and accuracy benchmarks tend to look fine until the one that bites. For PII detection a missed span is a leak, so prefer F16 when you can afford it (it is the reference precision) and treat Q8_0 as a deliberate size/speed tradeoff for constrained hardware — ideally re-checked on your own data.

Architecture & conversion

gpt-oss-style sparse MoE (8 layers, d_model=640, 128 experts, top-4 routing; ~1.5B total / ~50M active per token), bidirectional banded attention (symmetric sliding window, attention sinks retained), interleaved (GPT-J) RoPE with YaRN (θ=150000, factor 32), o200k (o200k_base) tokenizer, and a 33-way token-classification head (score → cls.output). privacy-filter.cpp re-derives the YaRN truncate=false frequencies at load time (fed to ggml_rope_ext as freq_factors) so the GGUF is interchangeable across runtimes.

Label space

O plus B-/I-/E-/S- for each of 8 categories (1 + 8×4 = 33): account_number, private_address, private_date, private_email, private_person, private_phone, private_url, secret. The ordered id2label table is embedded in the GGUF (classifier.output_labels).

Limitations & intended use

Identical to the source model: trained for high-throughput data sanitization, not a substitute for legal/compliance review, and not a clinical PHI model. Use it as one tier behind deterministic regex pre-filters and human review. For multilingual text, prefer the multilingual fine-tune.

License

Apache-2.0, inherited from openai/privacy-filter.

Credits & citation

Model by OpenAI (openai/privacy-filter). GGUF conversion and runtime support (privacy-filter.cpp) by the LocalAI project. Please cite OpenAI per the source card.