ambrosehui/flan-t5-small-safety-judgment
flan-t5-small-safety-judgment
1. Project Overview
This repository contains a fine-tuned version of the Flan-T5-Small model, specifically optimized for detecting Prompt Injection attacks. The model acts as a security guardrail, classifying incoming user prompts as either Safe or Injection.
2. Model Performance
The following metrics were derived from a test dataset evaluation. The model focuses on high sensitivity to ensure malicious inputs are intercepted before reaching the core LLM.
Confusion Matrix
Key Metrics
- Accuracy: 82.69%
- Precision: 82.53%
- Recall (Sensitivity): 86.12%
- F1-Score: 84.29%
Analysis
The model demonstrates a strong Recall of 86.12%, which is critical for security tasks. While there is a False Positive rate of ~21% (193 cases), this "strict" posture is often preferred in production environments to prevent bypasses, even at the cost of occasional over-blocking.
3. Disclaimers & Bias Statement
Disclaimer
Important: This model is provided "as-is" for research and supplementary security purposes. Prompt injection is an evolving adversarial field; therefore, this model should not be your only line of defense. It is highly recommended to use this in conjunction with output filtering, structural delimiters, and least-privilege API permissions.
Dataset & Potential Bias
- Pattern Bias: The training data may be biased toward common injection strings (e.g., "Ignore previous instructions"). It may underperform on novel, obfuscated, or multi-step "jailbreak" attempts.
- Linguistic Bias: The model was primarily trained on English-language datasets. Performance on multilingual prompts or code-heavy inputs (e.g., Python/SQL injection) may vary and should be validated separately.
- Contextual False Positives: The model may flag legitimate administrative commands as injections if the phrasing closely mimics adversarial patterns.
Technical Limitations
As a "Small" model (approx. 80M parameters), Flan-T5-Small may lack the deep semantic nuance of larger models like T5-Base or Large, but it offers significantly lower latency and resource consumption for real-time monitoring.