Owlchemy/privacy-filter-nemotron-v2-GGUF

privacy-filter-nemotron-v2-GGUF

GGUF conversion of `OpenMed/privacy-filter-nemotron-v2` for the `privacy-filter.cpp` runtime, with the Mixture-of-Experts weights requantized to Q8_0.

This is the Nemotron label schema checkpoint: 221 BIOES classes over 55 fine-grained PII categories. Its sibling, `Owlchemy/privacy-filter-multilingual-v2-GGUF`, uses the multilingual schema with 217 classes.

Files

How it was produced

Converted from OpenMed/privacy-filter-nemotron-v2 at revision 968247329d18998cc7d5338b941b52f1b2a9abd9, using the conversion scripts from privacy-filter.cpp at commit 735a6c28607ee82afc3a670383f41b55266a3b9a.

Step 1 - Hugging Face checkpoint to f16 GGUF, with `scripts/convert.py`:

hf download OpenMed/privacy-filter-nemotron-v2 --local-dir pf-src

python scripts/convert.py \
    --model pf-src \
    --outfile pf-nemotron-v2-f16.gguf \
    --outtype f16

convert.py is self-contained: it reads config.json, model.safetensors and tokenizer.json directly and does not depend on llama.cpp or its conversion script. The f16 output is 2,817,465,120 bytes.

Step 2 - requantize the expert weights to Q8_0, with `scripts/requant_q8.py`:

python scripts/requant_q8.py \
    --in pf-nemotron-v2-f16.gguf \
    --out pf-nemotron-v2-q8.gguf

Environment. torch (CPU build is sufficient), safetensors, and gguf>=0.10, as pinned by scripts/requirements.txt. Roughly 30 seconds for the conversion and 60 seconds for the requant on a CPU-only machine.

The upstream project's own `ci.yml` converts checkpoints to f16 and f32 on its nightly tier and gates them against reference fixtures; it does not run the Q8_0 requant, so step 2 above is the part that is not covered by upstream CI.

Architecture

The checkpoint is the openai/privacy-filter body (a gpt-oss style MoE) fine-tuned on the Nemotron PII mix and repurposed as a bidirectional token classifier. As recorded in the GGUF metadata:

Notes

• —The requant is partial. Only tensors matching ffn_gate_exps.weight, ffn_up_exps.weight and ffn_down_exps.weight (the MoE expert weights) become Q8_0. Every other tensor is copied verbatim and stays f16. This is what makes the file 1.64 GB rather than the 2.8 GB f16 intermediate, and it is the configuration the int8 mul_mat_id kernel is fastest on.
• —`general.file_type` still reports `f16` (value 1). requant_q8.py copies metadata fields verbatim, so the declared file type is not updated to reflect the Q8_0 expert tensors.
• —`gguf-py` cannot reopen these files. The reader raises KeyError: Duplicate GGUF.version already in list. The cause is not a duplicate in the file: requant_q8.py copies the reader's synthetic fields into the output as real metadata, so the file carries literal GGUF.version, GGUF.tensor_count and GGUF.kv_count entries that collide with the ones gguf-py injects when reading. llama.cpp and privacy-filter.cpp both load the files normally. The published LocalAI-io q8 files share this quirk.
• —The label schema differs from the multilingual checkpoint. Class names are snakecase here (`B-accountnumber) where the multilingual checkpoint uses uppercase concatenated names (B-ACCOUNTNAME`), and the two have different class counts. Do not swap one for the other without re-checking the label mapping.