it4lia/PhishingEmailCuratedDatasets_Cleaned

1---2language:3- en4license: cc-by-4.05pretty_name: Phishing Email Curated Cleaned6task_categories:7- tabular-classification8tags:9- cybersecurity 10- phishing 11- phishing-detection 12- email-classification 13- text-classification 14- benchmark 15- tabular 16- ai-ready 17- nlp 18size_categories:19- 100K<n<1M20---21 22# Phishing Email Curated Cleaned23 24Phishing Email Curated Cleaned is a cleaned and AI-ready version of the original **Phishing Email Curated Datasets** by Champa, Rabbi and Zibran (2024), an aggregation of 11 heterogeneous email corpora released on Zenodo for benchmarking phishing email detection with machine learning. 25The original collection aggregates emails from public corpora spanning 1995–2022 (CEAS-08, Ling-Spam, Enron, Nazario phishing corpus, Nigerian Fraud, SpamAssassin, TREC-05/06/07 plus the extended Nazario_5 and Nigerian_5 splits). The cleaned release preserves the multi-source nature of the data while making it easier to load, more reproducible, and directly usable for supervised classification, NLP-based detection, and cross-source robustness studies.26 27Compared with the original source asset, this release normalizes the heterogeneous schemas of the 11 source CSVs, applies a label-rescue policy on records with invalid labels, removes duplicate emails, audits constant and non-finite features, and exports a unified labeled dataset in a stable, memory-mapped, ML-ready format. Each sample is represented as a fixed-length numerical vector of 53 features hand-engineered from the raw email text (subject, body, headers, URL signals), while the original textual fields are preserved separately as metadata for audit and traceability.28 29## Original datasets30 31This cleaned release is derived from multiple phishing and legitimate email datasets:32- **Original name:** Phishing Email Curated Datasets33- **Original provider:** Champa, Rabbi & Zibran (Idaho State University) — released on Zenodo34- **Original papers:**  A. I. Champa, M. F. Rabbi, and M. F. Zibran (2024). *Why phishing emails escape detection: A closer look at the failure points*35                        A. I. Champa, M. F. Rabbi, and M. F. Zibran (2024). *Curated datasets and feature analysis for phishing email detection with machine learning.*36- **Original DOI:** https://doi.org/10.5281/zenodo.833969137- **Original project/repository:** https://zenodo.org/records/833969138Please cite the original datasets and related publications when using this cleaned release in research.39 40## Files41 42| File | Description |43|---|---|44| `email_clean.npz` | Index file with row/feature counts and file references |45| `email_clean_X.npy` | Feature matrix (`float32`), NumPy `.npy` array, shape `(181907, 53)` | 46| `email_clean_y.npy` | Label vector (`int32`), `0 = legitimate`, `1 = phishing` |47| `email_clean_metadata.parquet` | Per-sample metadata including source dataset fields and quality flags |48| `canonical_manifest_final.json` | Versioned manifest with checksums and artifact references |49| `email_curated_cleaned_dataset.ipynb` | Exploration and usage notebook |50 51## What’s in the dataset?52 53This cleaned release contains a fully labeled phishing email classification dataset built from multiple curated sources.54 55### Labeled split56 57- **181,907 labeled samples**58- **53 numerical features**59- feature dtype: `float32`60- label dtype: `int32`61- labels:62  - `0` = legitimate63  - `1` = phishing64 65The dataset combines phishing and legitimate emails originating from multiple heterogeneous corpora.66 67### Feature representation68 69Samples are represented as **fixed-length numerical feature vectors** derived from structured email-related characteristics.70 71The feature space includes information related to:72 73- email structure74- sender-related properties75- subject-related properties76- URL indicators77- attachment-related characteristics78- textual and statistical indicators79- phishing-related heuristics80 81The original source datasets may also contain raw email text or additional metadata fields. These are preserved in the metadata parquet file when available but excluded from the numerical ML feature matrix.82 83### Multi-source harmonization84 85One of the main goals of this cleaned release is the harmonization of heterogeneous phishing email corpora into a unified and reproducible tabular benchmark.86 87The pipeline standardizes:88 89- labels90- metadata structure91- feature schema92- source attribution93- duplicate handling94 95across all original datasets.96 97## Cleaning summary98 99This release is the output of a quality-control and harmonization pipeline applied to the original phishing email corpora.100 101Main processing steps:102 1031. **Duplicate removal** using canonical email identifiers1042. **Metadata standardization**1053. **Cross-source schema harmonization**1064. **Feature integrity validation**1075. **Missing-value and non-finite auditing**1086. **Manifest generation** for reproducibility and integrity checks109 110Summary of the main changes:111 112- **35,297 duplicate samples removed**113- **0 constant features dropped**114- **0 rows removed due to NaN/Inf/non-finite values**115- final dataset shape: **181,907 × 53**116 117No label conflicts were found during duplicate analysis.118 119## File structure120 121```text122Email_Curated_cleaned/123├── email_clean.npz124├── email_clean_X.npy125├── email_clean_y.npy126├── email_clean_metadata.parquet127└── canonical_manifest_final.json128```129 130The `.npz` index stores `_rows` and `_features` for reliable loading.  131The feature matrix is stored as a NumPy `.npy` array and can be loaded directly with `np.load(...)`.132 133## Requirements134 135To run the quickstart examples, install the minimum required dependencies:136 137```bash138pip install numpy pandas pyarrow139```140 141For notebook-based exploration and basic visualization, you may also install:142 143```bash144pip install jupyter matplotlib seaborn scikit-learn145```146 147## Quickstart148 149This example loads the labeled Phishing Email Curated Cleaned dataset and checks that features, labels, and metadata are consistent and ready for supervised use.150 151```python152import numpy as np153import pandas as pd154 155idx = np.load("email_clean.npz", allow_pickle=True)156n_rows = int(idx["_rows"])157n_features = int(idx["_features"])158 159X = np.load("email_clean_X.npy")160y = np.load("email_clean_y.npy")161meta = pd.read_parquet("email_clean_metadata.parquet")162 163print(164    f"Dataset: {X.shape[0]} samples, {X.shape[1]} features | "165    f"labels: {y.shape[0]} | metadata columns: {meta.shape[1]}"166)167 168assert X.shape == (n_rows, n_features)169assert X.shape[0] == len(y) == len(meta)170assert set(np.unique(y)) == {0, 1}171 172print("Unique labels:", np.unique(y))173print("Metadata columns:", meta.columns.tolist())174print("All checks passed.")175```176 177## Notebook178 179The repository also includes an exploration notebook in `.ipynb` format, designed to provide additional context on the cleaned dataset, its structure, and its main analytical use cases.180 181The notebook can be used to:182 183- inspect the labeled split, its 53 features and its 11-source provenance 184- explore feature distributions, sparsity profile and per-class statistics 185- validate dataset consistency end-to-end 186- review two example use cases: feature importance with Random Forest and full model evaluation (precision/recall, confusion matrix, ROC curve)187 188To open it locally, run:189 190```bash191jupyter notebook email_curated_cleaned_dataset.ipynb192```193 194or, if you use JupyterLab:195 196```bash197jupyter lab email_curated_cleaned_dataset.ipynb198```199 200Make sure to open the notebook from the dataset root directory so that relative file paths resolve correctly.201 202## Typical use cases203 204Phishing Email Curated Cleaned supports:205 206- binary phishing email detection 207- benchmarking of tabular ML pipelines 208- feature importance and ablation analysis on lexical, structural and URL-based signals 209- cross-source generalisation studies (train on N corpora, test on a held-out source) 210- exploratory data analysis on email content and metadata 211- failure-point analysis: studying why some phishing emails escape ML detection212 213The accompanying notebook includes dataset loading, exploratory analysis, and example use cases focused on phishing email classification and feature evaluation.214 215## Notes and limitations216 217This is a structured phishing email dataset focused on engineered numerical features.  218The cleaned release does not distribute raw malicious email attachments or executable payloads.  219The dataset is derived from heterogeneous public corpora and may reflect source-specific collection biases.  220The dataset is not intended for temporal phishing analysis.  221Results obtained on this benchmark should not be over-generalized to all phishing ecosystems without additional validation.  222The dataset is intended for defensive research, benchmarking, and education.223 224## License225 226This cleaned release is derived from the original dataset. The original data files are associated with the CC BY 4.0 License. Please verify that your downstream redistribution and reuse remain aligned with the original source dataset.227 228## References229 230If you use this dataset, please cite the original Phishing Email Curated dataset release:231 232@inproceedings{champa2024phishing,233  title={Why Phishing Emails Escape Detection: A Closer Look at the Failure Points},234  author={Champa, Arifa I and Rabbi, Fazle and Zibran, Minhaz F},235  booktitle={2024 12th International Symposium on Digital Forensics and Security (ISDFS)},236  pages={1--6},237  year={2024},238  organization={IEEE}239}240 241@inproceedings{champa2024curated,242  title={Curated Datasets and Feature Analysis for Phishing Email Detection with Machine Learning},243  author={Champa, Arifa I and Rabbi, Md Fazle and Zibran, Minhaz F},244  booktitle={3rd IEEE International Conference on Computing and Machine Intelligence (ICMI)},245  pages = {1--7},246  year={2024}247}248 249## APA:250 251Arifa Islam, C. (2023). Phishing Email Curated Datasets [Data set]. Zenodo. https://doi.org/10.5281/zenodo.8339691252 253## Contacts254 255Shared by: ACN