thhepu/malicious-email-scorer
Malicious Email Scorer
A production email threat analysis system that scores emails for malicious indicators using a two-layer cascade pipeline (deterministic structural heuristics followed by semantic AI classifiers) and returns a risk score (0-100), a verdict, and explainable findings.
Tech Stack: Python 3.10+ | FastAPI | PyTorch | HuggingFace Transformers | BeautifulSoup4 | Google Apps Script
Table of Contents
- System Flow & Layer Architecture
- Software Architecture & Design Patterns
- Product Philosophy & Approach
- Evaluation
- System Advantages & Limitations
- Local Development
- Deployment
- Future Work
- Retrospective: What Would I Do Differently?
System Flow & Layer Architecture
When a user opens an email in Gmail, the add-on extracts the email's metadata and body, sends it to the FastAPI backend, and gets back a scored verdict. The backend runs a two-layer cascade: fast deterministic checks first, semantic AI classifiers second (only when needed).
USER OPENS EMAIL IN GMAIL
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ Gmail Add-on (Client) │
│ Google Apps Script · Triggers on email open │
│ Extracts: subject, sender, body_html, body_text, attachments │
└──────────────────────────┬──────────────────────────────────────────┘
│ HTTPS POST /analyze
│ X-API-Key authentication
▼
┌─────────────────────────────────────────────────────────────────────┐
│ FastAPI Backend (API Layer) │
│ Pydantic validation · 1 MB body cap · Auth middleware │
└──────────────────────────┬──────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ AnalysisManager (Orchestrator) │
│ │
│ ┌───────────────────────────────────────────────────────────────┐ │
│ │ LAYER 1 - Structural Heuristics (< 40 ms) │ │
│ │ Deterministic, rule-based, fully explainable │ │
│ │ │ │
│ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────────────┐ │ │
│ │ │ Hidden │ │ URL │ │ Prompt Injection │ │ │
│ │ │ Content │ │ Analysis │ │ Pattern Detection │ │ │
│ │ └──────────────┘ └──────────────┘ └──────────────────────┘ │ │
│ │ ┌──────────────┐ ┌──────────────┐ │ │
│ │ │ Structural │ │ Attachment │ │ │
│ │ │ Anomalies │ │ Surface │ │ │
│ │ └──────────────┘ └──────────────┘ │ │
│ │ │ │
│ │ Score >= 80 ──────────────────────────────> SHORT-CIRCUIT │ │
│ └──────────────────────────┬────────────────────────────────────┘ │
│ │ Score < 80 │
│ ▼ │
│ ┌───────────────────────────────────────────────────────────────┐ │
│ │ LAYER 2 - Semantic AI Classifiers (~200 ms) │ │
│ │ Parallel transformer inference │ │
│ │ │ │
│ │ ┌──────────────────────┐ ┌──────────────────────────────┐ │ │
│ │ │ Phishing Detector │ │ Prompt Injection Detector │ │ │
│ │ │ DistilBERT │ │ DeBERTa-v3-base │ │ │
│ │ │ (human-targeted) │ │ (agent-targeted) │ │ │
│ │ └──────────┬───────────┘ └───────────────┬──────────────┘ │ │
│ │ └──────────┬───────────────────┘ │ │
│ │ ▼ │ │
│ │ Weighted Probability Voting (75% / 25%) │ │
│ │ Language-aware confidence dampening │ │
│ │ Single-model caps (score <= 65, conf <= 0.65) │ │
│ └──────────────────────────┬────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌───────────────────────────────────────────────────────────────┐ │
│ │ Score Fusion & Agreement Gate │ │
│ │ │ │
│ │ effective_score = raw_score x confidence │ │
│ │ fused_score = max(effective_scores) │ │
│ │ │ │
│ │ L1 clean (< 15) + L2 high --> Cap L2 at 55 (suspicious) │ │
│ │ L1 mid (15-40) + L2 high --> Linear interpolation cap │ │
│ │ L1 high (>= 40) + L2 high --> L2 uncapped │ │
│ │ L1 high (>= 80) --> Skip L2 entirely │ │
│ └──────────────────────────┬────────────────────────────────────┘ │
│ │ │
└─────────────────────────────┼───────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────────────────┐
│ Final Verdict │
│ │
│ Score 0-100 · Verdict: clean / suspicious / malicious │
│ Per-layer findings · Fusion traceability metadata │
└─────────────────────────────────────────────────────────────────────┘How the Layers Interact
The system is built as a layered cascade. Layer 1 runs first and catches most common threats with deterministic rules. If L1 evidence is strong enough (score >= 80), it short-circuits the pipeline: L2 never runs, saving the inference cost entirely.
When L1 signals are inconclusive (score < 80), Layer 2 kicks in and runs two transformer classifiers in parallel. Their outputs are fused via weighted probability voting.
The Agreement Gate is the key safety mechanism between the two layers. It solves a real problem with NLP classifiers: they often produce confident predictions on completely benign inputs. When L1 finds no structural evidence of anything suspicious (score < 15), L2's effective score is hard-capped at 55, which keeps the verdict at suspicious at most. This means the AI models alone can never push a verdict to malicious without structural backup from L1.
Layer 1 - Structural Heuristics
Five specialized detectors analyze the email's structure, each producing typed HeuristicTrigger findings with severity levels and score contributions:
Scoring: decay + diversity. The scoring algorithm prevents any single detector from running up the score on its own, while rewarding variety of evidence. Repeated triggers from the same detector contribute less and less (decay factor: 0.3), but triggers from different detectors stack at full value. So an email with hidden content AND a URL mismatch AND prompt injection patterns scores much higher than one with ten hidden elements and nothing else.
Trusted sender discount. When a sender's domain appears in the Tranco Top 1M popularity list, cosmetic heuristics (hidden content, structural anomalies) are discounted by 80%. Dangerous signals like URL manipulation, prompt injection, and suspicious attachments are never discounted, because legitimate senders can be compromised too.
Layer 2 - Semantic AI Classifiers
Two transformer models run in parallel, each targeting a different threat type:
Outputs are combined via weighted probability voting (75% phishing / 25% prompt injection). Several safeguards prevent overconfident single-model verdicts:
- Single-model cap: if only one model flags positive, its score is capped at 65 and confidence at 0.65. Both models need to agree for a high-confidence malicious verdict.
- Language-aware dampening: both models are English-oriented. For non-Latin or very short text, confidence is scaled down (Latin: 1.0x, mixed: 0.6x, Hebrew: 0.25x, short text: 0.45x, empty: 0.0x).
- Head-tail truncation: long emails are truncated using a 67/33 head-tail split, keeping both the opening hook and the closing call-to-action, which are the two most useful regions for phishing detection.
Software Architecture & Design Patterns
The system uses a contract-based, registry-driven architecture that separates concerns cleanly and makes extension straightforward.
┌──────────────────────────────────────────────────────────────────────┐
│ FastAPI Router │
│ POST /analyze endpoint │
│ Validates input (Pydantic) · Builds EmailPayload · Calls Manager │
│ Accepts enable_l1 / enable_l2 toggles from client │
└──────────────────────────┬───────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────┐
│ AnalysisManager │
│ (Orchestrator + Registry) │
│ │
│ register(analyzer) Analyzers execute in registration order │
│ run(payload, skip) Filters by skip_layers, cascades, fuses │
│ │
│ Depends ONLY on: BaseAnalyzer protocol + AnalysisResult contract │
│ Imports NOTHING from L1 or L2 internals │
└─────────┬──────────────────────────────────────────┬─────────────────┘
│ │
▼ ▼
┌─────────────────────────┐ ┌──────────────────────────────┐
│ L1HeuristicsAnalyzer │ │ L2SemanticAnalyzer │
│ (Adapter) │ │ (Adapter) │
│ │ │ │
│ Wraps run_layer_one() │ │ Wraps run_layer_two() │
│ confidence = 1.0 │ │ Weighted voting + capping │
│ (deterministic) │ │ Language dampening │
└─────────┬───────────────┘ └──────────────┬───────────────┘
│ │
│ Both implement the same contract: │
│ │
│ ┌──────────────────────────────────┐ │
└───>│ BaseAnalyzer (Protocol) │<──┘
│ │
│ @property layer -> str │
│ async analyze(EmailPayload) │
│ -> AnalysisResult │
└──────────────────────────────────┘Plug & Play Design
The architecture is built around four principles that make the system easy to extend:
1. Protocol-Based Contracts (No Inheritance Required)
BaseAnalyzer is a typing.Protocol with @runtime_checkable. Analyzers don't inherit from a base class - they just need to match the interface shape. Any object with a layer property and an async analyze(EmailPayload) -> AnalysisResult method qualifies. This removes coupling to an inheritance hierarchy and makes testing simple: stub any object with the right shape.
2. Registry-Based Orchestration
The AnalysisManager is completely layer-agnostic. To add a new analysis layer (e.g., a sandbox engine, an LLM reasoning layer, a reputation lookup), you just implement a class matching the BaseAnalyzer protocol and call manager.register(your_analyzer). The manager, fusion logic, API layer, and all existing analyzers need zero changes. The AnalyzerLayer identifier is an open-ended str subclass, so plugins can use any string as a layer ID without editing source code.
3. Adapters for Interface Translation
Adapters (L1HeuristicsAnalyzer, L2SemanticAnalyzer) translate each layer's internal output format into the standardized AnalysisResult interface. The underlying detection engines are never modified; adapters handle the translation. This means Layer 1 and Layer 2 can change their internal data models independently without breaking the pipeline contract.
4. Strict Dependency Isolation
- Layer 1 has zero ML dependencies: only
beautifulsoup4,rapidfuzz, and standard library modules - Layer 2 imports
transformersandtorch, loaded lazily at startup rather than at import time - The Manager imports neither layer's internals and depends only on the contracts package
This means L1 can be deployed without PyTorch, L2 models can be swapped without touching L1, and the manager can be tested with lightweight stubs that return hardcoded results.
Product Philosophy & Approach
Core Strategy: Two-Angle Threat Diagnosis
Modern email threats have evolved beyond what either rule engines or ML classifiers can reliably catch alone. The system was designed by studying popular email attack vectors and building detection that looks at threats from two angles:
a) Detecting specific, known attack vectors: CSS-based content hiding, URL spoofing via homoglyphs, typosquatting against known domains, prompt injection templates, dangerous attachments, and more.
b) Identifying general suspicious anomalies: unusual structural patterns, abnormal entropy, suspicious language tone, or behavioral signals that don't match any single known attack but collectively suggest risk.
Both layers pursue both of these goals. The difference is not what they look for, but how they do it. Layer 1 uses a deterministic, rule-based approach: pattern matching, regex, structural analysis, and distance metrics that produce fully explainable, reproducible results. Layer 2 uses a semantic, probabilistic AI approach: transformer models that interpret language meaning and generalize beyond hardcoded signatures, trading explainability for broader coverage of novel threats.
Special Focus: Social Engineering
The system puts particular focus on two social engineering categories that represent the fastest-growing threat vectors:
- Phishing (human-targeted): emails designed to trick human readers into clicking malicious links, revealing credentials, or transferring funds. Detected via both structural signals (URL mismatches, hidden content) and semantic analysis (DistilBERT classifier trained on phishing corpus).
- Prompt Injection (agent-targeted): a threat category that barely existed two years ago. As LLM-powered email assistants become common, attackers embed hidden instructions in emails to hijack the AI agent processing them. Detected via regex pattern matching (L1) and a dedicated DeBERTa-v3-base classifier (L2).
Performance Goals
The system is designed for real-time use inside a Gmail client. Users expect near-instant feedback when opening an email:
- Under 1 second for the vast majority of emails (L1-only short-circuit path handles obvious threats in under 40ms)
- Hard maximum of 3 seconds for edge cases requiring full L2 inference on CPU
- Per-analyzer timeout of 5 seconds with graceful degradation: the pipeline never hangs or crashes, even if a model fails
Evaluation
The system's thresholds and detection rules were not tuned by guesswork. They were validated against labeled datasets of real emails.
Methodology
The evaluation pipeline (scripts/evaluate_pipeline.py) takes a systematic approach to threshold tuning and performance measurement:
1. Dataset Preparation. JSONL datasets covering prompt injection, malicious intent (phishing), and benign emails are loaded with stratified sampling to preserve the positive/negative class ratio.
2. Full Pipeline Inference. Each sample is run through the complete cascade pipeline (L1 + L2 + fusion), capturing the fused score, verdict, per-layer scores, confidence values, short-circuit behavior, and latency.
3. Confusion Matrix Analysis. Results are compared against ground truth labels to compute true positives, false positives, true negatives, and false negatives. Per-class metrics (Precision, Recall, F1) show where the system works and where it breaks.
4. Threshold Grid Search. Rather than picking thresholds by hand, the pipeline sweeps across a grid of malicious thresholds (30.0 to 97.5, step 2.5) and suspicious thresholds (10.0 to malicious threshold, step 2.5). Each combination is scored for F1 while enforcing a recall floor of 0.85: configurations that drop too many malicious emails are automatically disqualified, since false negatives are more dangerous than false positives.
5. Held-Out Test Evaluation. The best threshold configuration is applied to a held-out test set to check that it generalizes.
Current Results
The recall is strong: the system catches nearly every threat. The precision is not where it needs to be. With a false-positive rate above 50%, the system flags too many benign emails as threats.
This comes from a few factors: the off-the-shelf transformer models in Layer 2 were not trained on email-specific data, the scoring thresholds were derived from a limited labeled corpus, and public phishing datasets are noisy and imbalanced in ways that may not perfectly match real-world distributions. The threshold grid search also optimizes for recall first, which pushes precision down as a side effect.
It is worth noting that it is difficult to know how well our evaluation dataset represents real-world email distributions. Still, these metrics clearly highlight the system's current high-sensitivity bias (catching almost all threats, but with a high false-positive rate) and point directly to where further tuning is needed.
Why Measurement Matters
Email security systems fail silently. Without real evaluation, there is no way to know whether a threshold change that reduces false positives also lets real phishing through. The Confusion Matrix approach makes these tradeoffs visible and measurable, turning threshold selection from guesswork into an engineering decision.
System Advantages & Limitations
Advantages
- Highly modular and extensible. The registry-based, contract-driven architecture makes the system easy to build, operate, maintain, and expand. Adding a new heuristic, swapping an AI model, or introducing an entirely new analysis layer requires no changes to existing code. This modularity also makes debugging and testing much easier, since each component can be validated on its own.
- User empowerment through explainability. The system doesn't just return a score and a label. Every verdict comes with detailed, human-readable findings that explain what was detected and why it matters. This gives users the information they need to understand the threat, evaluate the risk themselves, and learn to recognize similar attacks in the future.
Limitations
- Overly sensitive. The system currently leans toward paranoia, frequently flagging benign emails as threats. The 41% precision means more than half of flagged emails are false positives. While this is a deliberate tradeoff favoring safety over convenience, it degrades user trust over time. The architecture makes parameter tuning straightforward, so this is solvable with more data and iteration.
- AI "black box" in Layer 2. Because Layer 2 relies on transformer models (DistilBERT, DeBERTa-v3-base), there is no way to trace which specific words or patterns drove the model's prediction. L2 outputs a probability and a label, but unlike L1's transparent heuristic triggers, the reasoning behind it is opaque. This contrast between L1's full explainability and L2's opaque inference is a built-in limitation of the current design.
Local Development
Prerequisites
- Python 3.10+
- ~2 GB disk space for transformer model downloads (first run)
Setup
# Clone and enter the project
git clone <repository-url>
cd malicious-email-scorer
# Create and activate virtual environment
python -m venv venv
source venv/bin/activate # Linux/macOS
venv\Scripts\activate # Windows
# Install dependencies
pip install -r requirements.txt
# Set required environment variable
export API_KEY="your-secret-key"
# Start the development server
uvicorn app.main:app --reloadRunning Tests
# Full test suite
pytest
# Single module
pytest tests/test_hidden_content.py -v
# Single test
pytest tests/test_manager.py::test_confidence_weighted_fusion -vPipeline Evaluation
# Run end-to-end evaluation with threshold grid search
python scripts/evaluate_pipeline.pyDeployment
The backend is deployed to Hugging Face Spaces via auto-deploy from the main branch. The Dockerfile builds a Python 3.11 image, exposes port 7860, and runs a single uvicorn worker (single-worker is required because L2 models are module-level singletons).
The Gmail Add-on frontend is deployed separately via Google Apps Script (clasp push) and calls the Hugging Face Spaces URL.
API Endpoints
POST /analyze- Main analysis endpoint (requiresX-API-Keyheader)GET /rules- List active detection rules and their metadataGET /health- Liveness check (returns model load status)GET /docs- Swagger UI for interactive testing
Future Work
If I had more time, these are the directions I would take:
- Improve precision through fine-tuning. The single highest-impact investment would be collecting a larger, more carefully curated labeled dataset and fine-tuning the L2 models on actual email data. Paired with threshold optimization that targets F1 directly (rather than just enforcing a recall floor), this would substantially improve precision without giving up meaningful recall.
- Better AI models for email context. The current L2 classifiers (DistilBERT, DeBERTa-v3-base) were trained on short prompts and general text. Finding or fine-tuning models specifically trained on modern email structures, including HTML, headers, multi-part bodies, and thread context, would improve prompt injection and phishing detection accuracy.
- Granular toggle controls. The system currently supports toggling entire layers (L1/L2) on or off. The registry pattern already supports finer control; the next step is exposing per-heuristic and per-model toggles, so users could disable specific detectors (e.g., skip attachment analysis for internal emails) without affecting the rest of the pipeline.
- Data-driven scoring weights. The current scoring algorithm uses hand-picked conventions (decay factors, severity mappings, layer weights). Replacing these with optimized weights learned from labeled data would better calibrate the relationship between raw signals and final verdicts.
- Broader deployment options. The system currently ships as a Gmail Add-on. Expanding to Outlook plugins, a standalone web interface, or an API-first product would make the tool accessible to more users and workflows.
Retrospective: What Would I Do Differently?
Building this system taught me some real lessons about the gap between ML models on benchmarks and ML models in production:
1. Invest more heavily in L1 heuristics, rely less on L2 AI models.
In practice, the transformer classifiers turned out to be very prone to hallucinations and out-of-distribution noise when processing real-world emails. Emails contain boilerplate HTML, legal disclaimers, marketing templates, and multi-language content that look nothing like the short, clean text these models were trained on. The result was overconfident predictions on benign inputs, which is exactly the failure mode the Agreement Gate was built to contain. If I were starting over, I would put significantly more effort into expanding the deterministic heuristic coverage and treat L2 as a narrow supplement rather than a co-equal layer.
2. Establish scoring weights and thresholds through data from day one.
The initial scoring system was built on hand-picked conventions: severity levels, decay factors, and threshold boundaries chosen based on reasoning about what "should" indicate malice. While these conventions were reasonable, they required multiple rounds of recalibration once real evaluation data revealed gaps between intuition and actual score distributions. Starting with a data-first approach, collecting labeled samples before writing the scoring logic and then deriving weights from observed distributions, would have produced a better-calibrated system faster and with less rework.