faizhuda/priormail-phishing

Model Card — PriorMail Phishing Detector v1.0

Model ID: phishing/v1.0 Owner: Faiz (PJK-GM095) Date: 2026-06-13 Status: Eval gates passed — pending export approval from Insan + Syafiq

What This Model Does

Binary email classifier: legit (0) vs phishing (1).

Used by prior-mail-backend to flag potentially malicious emails before they reach the user's inbox. Runs at inference time on every incoming email.

Base Model

bert-base-multilingual-cased (Google, 178M params)

Chosen over IndoBERT because the primary training data is English-dominant (~98% EN). mBERT handles English well and retains multilingual capability for future Indonesian phishing data.

Training Data

Total after deduplication: ~38,000 rows Train / Val / Test split: 75 / 12.5 / 12.5 (stratified by label)

Deduplication applied on the first 200 chars of body before splitting to prevent leakage.

Note: Dataset is almost entirely English. Indonesian phishing emails are not well-represented. Performance on Indonesian-language phishing may be lower — to be addressed in v1.1 with internal labeled data.

Training Config

See `configs/phishing_v1.yaml` for the full reproducible config.

Key hyperparameters:

Evaluation Results

Threshold Selection (Validation Set)

Threshold selected on val set to maximize precision while keeping recall ≥ 0.95.

Threshold: 0.95

Final Results (Test Set, n=5,722)

Confusion Matrix (Test Set)

False negatives (missed phishing): 32 out of 1,050 — 3.0% miss rate. False positives (false alarms): 79 out of 4,672 — 1.7% false alarm rate.

Input / Output Format

Input (assembled by src/data/preprocess.build_phishing_input()):

FROM: {sender_email} [SEP] SUBJECT: {subject} [SEP] BODY: {body}

If sender or subject are unavailable (e.g. plain-body-only sources), those fields are empty strings. The model still performs well on body-only input — the HF training data was ~98% body-only.

Output:

• —Raw: softmax probability for class 1 (phishing), range [0, 1]
• —Final label: phishing if probability ≥ 0.95, else legit
• —Threshold stored in threshold.json alongside the checkpoint

Known Limitations

• —Language: Trained primarily on English data. Lower recall expected on Indonesian-language phishing. Mitigated by mBERT's multilingual pretraining, but not validated.
• —Sender signal weak: ~98% of HF phishing training samples had no RFC 2822 headers — sender email is not a reliable feature for this version. Enron legit samples do have sender headers, which may introduce a small asymmetry.
• —Adversarial robustness: Not evaluated against adversarial phishing (obfuscated URLs, lookalike domains, Unicode tricks). Planned for v1.1.
• —Domain shift: Trained on public phishing corpora. Real-world phishing targeting Indonesian corporate users may have different patterns.

Artifacts

Eval artifacts (in eval/results/phishing/): | File | Description | |---|---| | eval_report.json | Full test set metrics + confusion matrix + latency |

Changelog