agentlans/mdbr-leaf-mt-prompt-safety

mdbr-leaf-mt-prompt-safety

A regression model for assessing English-language prompt safety. It evaluates user intent rather than just surface-level toxicity.

• —Output: Float [0, 1] (0 = unsafe, 1 = safe).
• —Primary Use: Guardrail for LLM applications.

Quickstart

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model_name = "agentlans/mdbr-leaf-mt-prompt-safety" 

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name).to(device)

text = "How do I blow up a balloon for a party?"
inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True).to(device)

model.eval()
with torch.no_grad():
    score = model(**inputs).logits.item()

print(f"Safety Score: {score:.4f}") # 0.8342

Examples

Training & Methodology

1. 1.Pre-training: Masked Language Modeling (MLM) on prompt text.
2. 2.Fine-tuning: Regression against safety scores from the agentlans/prompt-safety-scores dataset.

Hyperparameters

• —Learning Rate: 5e-05
• —Batch Size: 8
• —Epochs: 20
• —Optimizer: AdamW (Fused)
• —Use Liger kernel

Performance (Regression)

Limitations

• —No Rationale: Provides a score without explaining the specific safety violation.
• —No Context: Evaluates single prompts; ignores conversational history.
• —Fixed Policy: Safety criteria are fixed to training data and cannot be adjusted via prompts.
• —Supportive Tool: Designed to complement, not replace, human moderation or broader content filters.

Licence

Apache 2.0