it4lia/EMBER_cleaned

EMBER Cleaned

EMBER Cleaned is a cleaned and AI-ready version of the original EMBER (Endgame Malware Benchmark for Research) dataset, a widely used benchmark for static malware detection on Windows Portable Executable (PE) files.

The original EMBER dataset was introduced by Endgame / Elastic as an open benchmark for machine-learning-based malware detection using only static PE-derived features, without executing binaries. This cleaned release preserves that purpose while making the dataset easier to load, more reproducible, and more directly usable for downstream experimentation.

Compared with the original source asset, this release standardizes metadata, removes duplicate samples, drops constant features, and exports unlabeled samples into a dedicated split for semi-supervised workflows. Each sample is represented as a fixed-length numerical vector derived from PE structure and content, including header information, section statistics, imports, and histogram-based features.

Original dataset

This dataset is a cleaned derivative of the original EMBER benchmark:

• —Original name: EMBER (Endgame Malware Benchmark for Research)
• —Original provider: Endgame / Elastic
• —Original paper: Anderson, H. S., & Roth, P. (2018). EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models
• —Original DOI: https://doi.org/10.48550/arXiv.1804.04637
• —Original project/repository: https://github.com/elastic/ember

Please cite the original EMBER paper when using this cleaned release in research.

Files

What’s in the dataset?

This cleaned release contains the labeled portion of EMBER plus a separate unlabeled split.

Labeled split

• —799,838 labeled samples
• —approximately balanced between benign and malicious files
• —2,099 numerical features
• —feature dtype: float32
• —label dtype: int32
• —labels:
• —0 = benign
• —1 = malware

Unlabeled split

• —199,966 unlabeled samples
• —exported separately for semi-supervised workflows
• —same 2,099-dimensional feature space
• —not intended to be interpreted as benign or malicious ground truth

Feature representation

Samples are not raw executables. Each file is represented as a fixed-length static feature vector extracted from the original PE file. These features describe structural and statistical properties of the binary, such as:

• —PE headers
• —imported APIs / libraries
• —sections
• —byte-histogram-related information
• —entropy-related characteristics

Cleaning summary

This release is the output of a quality-control and standardization pipeline applied to the original EMBER artifacts.

Main processing steps:

1. 1.Duplicate removal using feature fingerprints
2. 2.Constant-feature filtering, reducing the feature space from 2,381 to 2,099
3. 3.Metadata standardization
4. 4.Missing-value normalization and quality flagging
5. 5.Label separation, exporting label = -1 samples into a dedicated unlabeled split
6. 6.Manifest generation for reproducibility and integrity checks

Summary of the main changes:

• —196 duplicate samples removed
• —282 constant features dropped
• —199,966 unlabeled samples exported separately
• —final labeled dataset shape: 799,838 × 2,099

File structure

EMBER_cleaned/
├── ember_clean.npz
├── ember_clean_X.npy
├── ember_clean_y.npy
├── ember_clean_metadata.parquet
├── ember_unlabeled.npz
├── ember_unlabeled_X.npy
├── ember_unlabeled_y.npy
├── ember_clean_metadata_unlabeled.parquet
└── manifest.json

The .npz index stores rows and features for reliable loading. The feature matrices are raw memmap-backed arrays and should be loaded with explicit dtype and shape.

Requirements

To run the quickstart examples, install the minimum required dependencies:

pip install numpy pandas pyarrow

For notebook-based exploration and basic visualization, you may also install:

pip install jupyter matplotlib seaborn scikit-learn

Quickstart

This example loads the labeled EMBER Cleaned split and checks that features, labels, and metadata are consistent and ready for supervised use.

import numpy as np
import pandas as pd
 
idx = np.load("ember_clean.npz", allow_pickle=True)
n_rows = int(idx["_rows"])
n_features = int(idx["_features"])
 
X = np.fromfile("ember_clean_X.npy", dtype=np.float32).reshape(n_rows, n_features)
y = np.load("ember_clean_y.npy")
meta = pd.read_parquet("ember_clean_metadata.parquet")
 
print(f"Dataset: {X.shape[0]} samples, {X.shape[1]} features | " f"labels: {y.shape[0]} | " f"metadata columns: {meta.shape[1]}")
 
assert X.shape[0] == len(y) == len(meta)
assert set(np.unique(y)) == {0, 1}
 
print("Unique labels:", np.unique(y))
print("Labeled Metadata Columns:", meta.columns.tolist())
print("All checks passed.")

The following example loads the unlabeled EMBER Cleaned split and checks that features and metadata are aligned for semi-supervised or exploratory use.

import numpy as np
import pandas as pd
 
idx_u = np.load("ember_unlabeled.npz", allow_pickle=True)
n_rows_u = int(idx_u["_rows"])
n_features_u = int(idx_u["_features"])
 
X_u = np.fromfile("ember_unlabeled_X.npy", dtype=np.float32)
assert X_u.size == n_rows_u * n_features_u, (
    f"Unexpected X_u size: got {X_u.size}, expected {n_rows_u * n_features_u}"
)
X_u = X_u.reshape(n_rows_u, n_features_u)
 
meta_u = pd.read_parquet("ember_clean_metadata_unlabeled.parquet")
 
print(f"Dataset: {X_u.shape[0]} samples, {X_u.shape[1]} features | " f" unlabeled split: {n_rows_u} samples | " f"metadata columns: {meta_u.shape[1]}")
 
assert X_u.shape == (n_rows_u, n_features_u)
assert len(meta_u) == n_rows_u
 
if "label_int" in meta_u.columns:
    print("Unlabeled metadata labels:", np.unique(meta_u["label_int"]))
    assert set(np.unique(meta_u["label_int"])) == {-1}
 
print("Unlabeled metadata columns:", meta_u.columns.tolist())
print("Unlabeled split loaded successfully.")

Notebook

The repository also includes an exploration notebook in .ipynb format, designed to provide additional context on the cleaned dataset, its structure, and its main analytical use cases.

The notebook can be used to:

• —inspect the labeled and unlabeled splits
• —explore metadata fields and label distributions
• —validate dataset consistency
• —review example analyses and downstream use cases

To open it locally, run:

jupyter notebook ember_cleaned_dataset.ipynb

or, if you use JupyterLab:

jupyter lab ember_cleaned_dataset.ipynb

Make sure to open the notebook from the dataset root directory so that relative file paths resolve correctly.

Typical use cases

EMBER Cleaned supports:

• —binary malware detection
• —benchmarking of tabular ML pipelines
• —feature importance analysis
• —semi-supervised learning using the separate unlabeled split
• —exploratory data analysis
• —representation learning and clustering

The accompanying notebook includes dataset loading, exploratory analysis, and example use cases focused on discriminative features and model evaluation.

Notes and limitations

This is a static-analysis dataset only. The cleaned release contains derived features, not raw PE binaries. The unlabeled split should not be treated as ground truth. Results on EMBER should not be over-generalized to modern malware without additional validation. The dataset is intended for defensive research, benchmarking, and education.

License

This cleaned release is derived from EMBER. The original EMBER data files are associated with the MIT License. Please verify that your downstream redistribution and reuse remain aligned with the original EMBER terms.

References

If you use this dataset, please cite the original EMBER paper:

@article{anderson2018ember, title={EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models}, author={Anderson, Hyrum S. and Roth, Phil}, journal={arXiv preprint arXiv:1804.04637}, year={2018} }

APA:

Anderson, H. S., & Roth, P. (2018). EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models. arXiv. https://doi.org/10.48550/arXiv.1804.04637

Contacts

Shared by: ACN