bbanany/qwen25-3b-korean-pii-qlora

Qwen2.5-3B Korean PII Candidate Classifier (QLoRA)

This repository contains a LoRA adapter for Qwen/Qwen2.5-3B-Instruct. It classifies one upstream NER candidate in the context of a Korean RAG answer as exactly PII or NOT_PII. It is an adapter-only repository; the Qwen base model is required for inference.

Evaluation

Greedy label generation was evaluated on the 1,000-record synthetic validation split.

Confusion counts were PII: TP=500, FP=0, FN=0 and NOT_PII: TP=500, FP=0, FN=0. The loss-based validation result was eval_loss=1.002679e-06.

These are in-distribution synthetic validation results. The split is profile-disjoint, but synthetic sentence/template fragments are shared with training data. No independent production or real-world test score is claimed. Do not interpret the perfect validation result as evidence of perfect production performance.

Training

• —Base model: Qwen/Qwen2.5-3B-Instruct
• —Method: 4-bit NF4 QLoRA with double quantization
• —LoRA: rank 16, alpha 32, dropout 0.05
• —Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• —Training loss: assistant answer tokens only
• —Maximum sequence length: 512
• —Training records: 8,000 (4,000 per label)
• —Validation records: 1,000 (500 per label)
• —Epochs: 3
• —Effective batch size: 16
• —Learning rate: 2e-4 with cosine decay
• —Optimizer updates: 1,500
• —Hardware: one NVIDIA A40, BF16 compute
• —Seed: 20260727

Detailed settings, package versions, and dataset hashes are in run_manifest.json. Raw evaluation artifacts are included in valid_generation_metrics.json, eval_results.json, and train_results.json.

Training dataset: `bbanany/step4_sllm_v2`

Inference

Install recent compatible versions of torch, transformers, peft, and accelerate, then set REPO_ID to this repository:

import json
import torch
from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

REPO_ID = "YOUR_NAMESPACE/YOUR_MODEL_REPO"

tokenizer = AutoTokenizer.from_pretrained(REPO_ID)
model = AutoPeftModelForCausalLM.from_pretrained(
    REPO_ID,
    torch_dtype=torch.bfloat16,
    device_map="auto",
).eval()

system_prompt = (
    "당신은 RAG 답변에서 상위 NER 단계가 추출한 후보를 검증하는 개인정보 이진 분류기입니다. "
    "답변 전체 문맥을 근거로 후보가 특정 자연인의 개인정보 속성으로 서술되면 PII, "
    "예시·선택지·일반 지식·기관이나 행사 자체에 관한 표현처럼 특정 자연인과 연결되지 않으면 "
    "NOT_PII로 판정하세요. 반드시 PII 또는 NOT_PII 중 하나만 출력하고 설명은 덧붙이지 마세요."
)
answer = "민원인 김민수 씨의 연락처 변경 요청을 접수했습니다."
candidate_text = "김민수"
start = answer.index(candidate_text)
payload = {
    "answer": answer,
    "candidate": {
        "text": candidate_text,
        "tag": "NAME",
        "start": start,
        "end": start + len(candidate_text),
    },
}
messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": json.dumps(payload, ensure_ascii=False, separators=(",", ":"))},
]
inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
).to(model.device)
with torch.inference_mode():
    output = model.generate(
        inputs,
        do_sample=False,
        max_new_tokens=8,
        eos_token_id=tokenizer.convert_tokens_to_ids("<|im_end|>"),
        pad_token_id=tokenizer.pad_token_id,
    )
print(tokenizer.decode(output[0, inputs.shape[1]:], skip_special_tokens=True).strip())

An equivalent runnable example is provided in inference_example.py.

Intended use and limitations

This model is intended as a downstream validation component for an upstream NER system. It is not a general-purpose PII detector and does not find spans by itself. It must receive the full answer, one candidate span, its offsets, and its entity tag in the training schema.

The training data is synthetic. Korean language variation, OCR noise, adversarial inputs, unseen entity categories, offset errors, and domain shift may reduce performance. Use an independent real-world test set and human review before deployment. Do not use model output as the sole basis for consequential privacy, legal, security, or access-control decisions.

Artifact integrity

MODEL_VERIFICATION.json records the adapter tensor inventory, metric consistency checks, and SHA-256 digests produced during release preparation.