thc1006/cyberpuppy-v6-bilingual
0
CyberPuppy v6 — Bilingual Text LoRA (LoRA-A)
中文網路霸凌偵測 v6 · Qwen3-8B + LoRA r=64 + 4-head + 雙語 + CNTP 對抗訓練 Updated v6 with doubled LoRA capacity (r=64) and 5-epoch training for stronger phonetic robustness on systematic homophonic perturbations. Part of dual-LoRA ensemble with v6-pinyin-lora.
What changed from v5
Note: Both v5.1 and v6 best checkpoints peak at epoch 1-2 — additional capacity helps downstream test sets (especially HED-COLD), not standalone dev F1.
Performance (dual-LoRA ensemble v6 + α=0.60 + dec_thresh=0.52)
PCR-ToxiCN remains world-class: v6.0's 0.7119 still exceeds published SOTA 0.672 by +3.99pt (vs v5.1.1's +4.42pt). Trade-off accepted for HED-COLD breakthrough.
Quick Start
import torch, re
from peft import PeftModel
from transformers import AutoModel, AutoTokenizer
from huggingface_hub import hf_hub_download
from pypinyin import pinyin, Style
device = torch.device("cuda")
dtype = torch.bfloat16
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B-Base")
def load_branch(repo_id):
base = AutoModel.from_pretrained("Qwen/Qwen3-8B-Base", torch_dtype=dtype, device_map=device)
model = PeftModel.from_pretrained(base, repo_id, subfolder="lora")
model.eval()
# heads.pt: nn.ModuleDict({task: nn.Linear(H, dim)}) state_dict
heads_state = torch.load(hf_hub_download(repo_id, "heads.pt"),
map_location=device, weights_only=False)["heads"]
W_tox = heads_state["toxicity.weight"].to(device=device, dtype=dtype) # (3, H)
b_tox = heads_state["toxicity.bias"].to(device=device, dtype=dtype) # (3,)
return model, W_tox, b_tox
model_t, W_t, b_t = load_branch("thc1006/cyberpuppy-v6-bilingual")
model_p, W_p, b_p = load_branch("thc1006/cyberpuppy-v6-pinyin-lora")
_HAN = re.compile(r"[㐀-䶿一-鿿豈-]")
def to_pinyin(text):
return " ".join(pinyin(ch, style=Style.NORMAL)[0][0] if _HAN.match(ch) else ch
for ch in text if ch.strip())
@torch.inference_mode()
def predict(text, alpha=0.60, dec_thresh=0.52):
enc_t = tok(text, return_tensors="pt", truncation=True, max_length=192).to(device)
enc_p = tok(to_pinyin(text), return_tensors="pt", truncation=True, max_length=192).to(device)
h_t = model_t(**enc_t).last_hidden_state[:, -1]
h_p = model_p(**enc_p).last_hidden_state[:, -1]
logits_t = (h_t @ W_t.T + b_t).float()
logits_p = (h_p @ W_p.T + b_p).float()
text_probs = logits_t.softmax(-1)
pinyin_probs = logits_p.softmax(-1)
# v6.0 strategy: α=0.60 geo-mean, no pinyin override, decision_thresh=0.52
ens = (text_probs ** alpha) * (pinyin_probs ** (1 - alpha))
ens = ens / ens.sum(-1, keepdim=True)
toxic_prob = (ens[:, 1] + ens[:, 2]).item()
return "toxic" if toxic_prob > dec_thresh else "none", toxic_prob
label, p = predict("你這個笨蛋，滾開！")
print(f"{label} (toxic_prob={p:.3f})") # toxicTraining Details
Limitations
- PCR-ToxiCN slightly lower than v5.1.1 (0.7119 vs 0.7162). For maximum real-world robustness, you can use α=0.5 + pinyin override 0.7 + thresh 0.48 instead, getting PCR 0.7283 but COLD drops to 0.8198.
- English input: Out of distribution.
- Cultural / semantic attacks: e.g., 動物園貓孝子, 淺草寺吧 — these require world knowledge, not phonetics. ~30-40 of PCR's 250 toxic samples remain hard.
- Numeric / letter substitution attacks (G8, S13, 64.5克黄金=250) are partially handled by training data exposure but not fully solved.
- Sarcasm: 「你真太6了」used sarcastically can be missed.
License
CC BY-NC-SA 4.0. Non-commercial research and educational use only.
Citation
@misc{cyberpuppy_v6_2026,
author = {Tsai, Hung-Che},
title = {CyberPuppy v6: Doubled-Capacity Dual-LoRA for Chinese Cyberbullying Detection},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/thc1006/cyberpuppy-v6-bilingual}}
}Related Models
- thc1006/cyberpuppy-v6-pinyin-lora — companion pinyin LoRA
- thc1006/cyberpuppy-v5-bilingual — previous version (still has higher PCR-ToxiCN)
- thc1006/cyberpuppy-v5-pinyin-lora — v5 pinyin LoRA
Contact & Takedown
- Author: Hung-Che Tsai (hctsai1006@cs.nctu.edu.tw)
- Takedown: Email above — removed within 7 days