Modelpublic

CrabInHoney/urlbert-tiny-v6

sourceHugging Faceapache-2.0updated 14d agoView on Hugging Face

1likes374downloads

Model Card

CrabInHoney/urlbert-tiny-v6

CrabInHoney/urlbert-tiny-v6 is a lightweight URL encoder based on the ModernBERT architecture, designed for malicious URL detection, phishing classification, and URL feature extraction.

With only 2.03M parameters and a hidden dimension of 128, the model executes very fast simply due to its small footprint and compact embedding size, while remaining competitive on detection tasks.

The base model was trained via knowledge distillation of hidden representations from an ensemble of multiple teacher models, which were intentionally trained specifically for this purpose. Please note that this model is designed to process and analyze URL strings exclusively.

The repository contains the base encoder alongside 10 pre-trained classification heads trained on various public cybersecurity datasets.

Key Specifications

Parameter	Value
Architecture	ModernBERT (`ModernBertModel`)
Parameters	2,033,408 (2.03M total; 0.98M backbone, 1.05M embeddings)
Hidden Dimension	128
Layers / Heads	6 hidden layers / 4 attention heads
Context Window	256 tokens
Vocabulary Size	8,193
Model Size	~7.76 MB (FP32) / 3.88 MB (FP16)
Average Tokens / URL	24.8 (Truncation rate: 0.093% at `max_len=256`)

Benchmark Comparison

The models were evaluated on the JPxxx/url-benchmark-dataset

The benchmark contains 3M URLs with a strict 9:1 benign-to-malicious ratio. Evaluation was conducted using site-aware 5-fold cross-validation, ensuring that no second-level domain or IP address appears in more than one fold to eliminate data leakage.

Rank	Model	Base Dim	AUPRC ↑	AUROC ↑	F1 ↑	TPR @ 0.1% FPR ↑	TPR @ 1% FPR ↑	FPR @ 95% TPR ↓
1	CrabInHoney/urlbert-tiny-v6	128	0.9732	0.9928	0.9302	0.8339	0.9337	0.0173
2	r3ddkahili/final-complete-malicious-url-model	768	0.9627	0.9901	0.9120	0.7855	0.9076	0.0335
3	kmack/malicious-url-detection	768	0.9626	0.9903	0.9137	0.7491	0.9123	0.0294
4	ealvaradob/bert-finetuned-phishing	1024	0.9451	0.9858	0.8843	0.6446	0.8658	0.0544
5	cybersectony/phishing-email-detection-distilbert_v2.4.1	768	0.9426	0.9855	0.8803	0.6670	0.8600	0.0645
6	CrabInHoney/urlbert-tiny-base-v4	192	0.9339	0.9826	0.8716	0.6523	0.8416	0.0844
7	CrabInHoney/urlbert-tiny-base-v3	192	0.8871	0.9720	0.8096	0.4817	0.7372	0.1531
8	CrabInHoney/urlbert-tiny-base-v2	192	0.8223	0.9567	0.7381	0.3657	0.6123	0.2218

Feature Extraction (Embeddings)

python

from transformers import AutoModel, AutoTokenizer
import torch

REPO = "CrabInHoney/urlbert-tiny-v6"

tok = AutoTokenizer.from_pretrained(REPO)
model = AutoModel.from_pretrained(REPO)

text = "http://example.com/login-verify-account"
inputs = tok(text, return_tensors="pt")

with torch.no_grad():
    out = model(**inputs)

print(out.last_hidden_state.shape)

Output:

text

torch.Size([1, 15, 128])

Important Note on Pooling: For optimal downstream performance, it is highly recommended to use a combination of CLS + MEAN pooling (concatenating the [CLS] token representation with the mean representation of all tokens). This strategy yields the most robust and useful signal for feature extraction.

Pretrained Classification Heads

Trained classification heads are stored in the heads/ directory:

Head Subfolder	Source Dataset	Classes	Accuracy	Test Loss
`phiusiil_phishing_url`	PhiUSIIL Phishing URL Dataset	2	99.86%	0.0833
`cybersectony_phishing_email_v2`	Cybersectony Phishing v2.0	2	99.83%	0.0841
`ealvaradob_phishing_dataset`	EA Phishing Dataset	2	99.40%	0.0965
`malicious_urls_4class`	Malicious URLs Dataset	4	99.26%	0.1556
`url_65lakh`	65 Lakh+ Labeled URLs	2	99.21%	0.1015
`phishbd_2026`	PhishBD_2026	2	97.27%	0.1364
`phishdestroy_destroylist`	PhishDestroy Destroylist	2	95.68%	0.1823
`kmack_phishing_urls`	KMack Phishing URLs	2	90.89%	0.2514
`snats_url_classifications_clean`	Snats URL Classifications	17	58.91%	1.4642
`weborganizer_topic_annotations`	WebOrganizer Topic Annotations	24	52.32%	1.7157

Multi-Head Classification Example

python

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

REPO = "CrabInHoney/urlbert-tiny-v6"

HEADS = [
    "cybersectony_phishing_email_v2",
    "ealvaradob_phishing_dataset",
    "kmack_phishing_urls",
    "malicious_urls_4class",
    "phishbd_2026",
    "phishdestroy_destroylist",
    "phiusiil_phishing_url",
    "snats_url_classifications_clean",
    "url_65lakh",
    "weborganizer_topic_annotations",
]

tok = AutoTokenizer.from_pretrained(REPO)
text = "http://paypal-secure-login.verify-account.com"

for head in HEADS:
    model = AutoModelForSequenceClassification.from_pretrained(
        REPO, subfolder=f"heads/{head}", trust_remote_code=True
    )
    inputs = tok(text, return_tensors="pt", truncation=True, max_length=model.config.max_length)

    with torch.no_grad():
        logits = model(**inputs).logits

    probs = torch.softmax(logits, dim=-1)[0]
    pred = probs.argmax().item()
    print(f"{head:35s} -> {model.config.id2label[pred]:15s} ({probs[pred]:.3f})")

Output:

text

cybersectony_phishing_email_v2      -> phishing_url    (0.984)
ealvaradob_phishing_dataset         -> phishing        (0.985)
kmack_phishing_urls                 -> phishing        (0.984)
malicious_urls_4class               -> phishing        (0.979)
phishbd_2026                        -> phishing        (0.985)
phishdestroy_destroylist            -> malicious       (0.984)
phiusiil_phishing_url               -> phishing        (0.985)
snats_url_classifications_clean     -> legal           (0.352)
url_65lakh                          -> malicious       (0.985)
weborganizer_topic_annotations      -> Finance & Business (0.485)