Modelpublic

bharath-0201/omni-shield-qwen2vl-pii-adapter

sourceHugging Faceapache-2.0updated 6mo agoView on Hugging Face

0likes4downloads

Model Card

Omni-Shield QLoRA Adapter for PII Detection

Fine-tuned adapter for Qwen2-VL-2B-Instruct trained on synthetic identity documents for PII detection and redaction.

Training details

—Base model: Qwen/Qwen2-VL-2B-Instruct
—Method: QLoRA (rank-16 LoRA adapters, 4-bit NF4 quantization)
—Training data: 2,000 synthetic ID cards (6 countries: AU, CA, DE, IN, UK, US)
—Teacher labels: Claude API (knowledge distillation via annotatewithclaude.py)
—Hardware: NVIDIA RTX 5060 8GB VRAM
—Training time: ~47 minutes, 3 epochs
—Optimizer: adamwbnb8bit

Supported document types

ID cards and driving licences from: Australia, Canada, Germany, India, UK, USA

PII categories detected

names, faces, signatures, phones, emails, dob, id_numbers, addresses, org_names, dates

Usage

python

from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from peft import PeftModel
import torch

processor = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-2B-Instruct")

base_model = Qwen2VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2-VL-2B-Instruct",
    load_in_4bit=True,
    device_map="auto",
    torch_dtype=torch.bfloat16,
)
model = PeftModel.from_pretrained(
    base_model,
    "bharath-0201/omni-shield-qwen2vl-pii-adapter"
)
model = model.merge_and_unload()
model.eval()

Key finding: label consistency

Fine-tuning teaches the model to emit structured LABEL | VALUE pairs consistently. The base model detects values correctly but frequently omits field labels, causing downstream pipeline failures. Fine-tuning fixes this without degrading value detection accuracy.

Example (German ID card): | Field | Base model output | Fine-tuned output | |-----------|---------------------------|---------------------------| | Name | Name \| Leonie Dörr | Name \| Leonie Dörr | | DOB | Date of Birth \| 01.08.1991 | Date of Birth \| 01.08.1991 | | ID Number | [empty] \| M898585166 | ID No. \| M898585166 | | Signature | Signature \| [none] | Signature \| [none] |

Citation

bibtex

@article{{omni-shield-2025,
  title={{Omni-Shield: Local-First Multi-Modal PII Redaction
         with Cryptographic Audit Trail}},
  author={{[Anonymous for review]}},
  year={{2025}}
}}

Paper

Submitted to PETS 2026. arXiv preprint forthcoming.