mjbommar/binary-30k-tokenized

1---2license: cc-by-4.03task_categories:4- other5tags:6- binary-analysis7- malware-detection8- executable-analysis9- binary-tokenization10- cybersecurity11- reverse-engineering12- program-analysis13- cross-platform14size_categories:15- 10K<n<100K16pretty_name: 'Binary-30K: A Large-Scale Multi-Platform Binary Dataset'17configs:18- config_name: default19  data_files:20  - split: train21    path: data/train-*22dataset_info:23  features:24  - name: file_id25    dtype: string26  - name: file_path27    dtype: string28  - name: file_name29    dtype: string30  - name: sha25631    dtype: string32  - name: md533    dtype: string34  - name: file_size35    dtype: int6436  - name: platform37    dtype: string38  - name: os_family39    dtype: string40  - name: os_version41    dtype: string42  - name: distribution43    dtype: string44  - name: is_malware45    dtype: bool46  - name: file_format47    dtype: string48  - name: architecture49    dtype: string50  - name: binary_type51    dtype: string52  - name: is_stripped53    dtype: bool54  - name: is_packed55    dtype: bool56  - name: is_signed57    dtype: bool58  - name: sections59    struct:60    - name: name61      list: string62    - name: size63      list: int6464    - name: entropy65      list: float3266  - name: num_sections67    dtype: int3268  - name: code_size69    dtype: int6470  - name: data_size71    dtype: int6472  - name: imports73    list: string74  - name: num_imports75    dtype: int3276  - name: exports77    list: string78  - name: num_exports79    dtype: int3280  - name: entropy81    dtype: float3282  - name: token_count83    dtype: int3284  - name: compression_ratio85    dtype: float3286  - name: unique_tokens87    dtype: int3288  - name: parse_status89    dtype: string90  - name: parse_warnings91    list: string92  - name: has_tokens93    dtype: bool94  - name: tokens95    list: int3296  splits:97  - name: train98    num_bytes: 2397436622699    num_examples: 29793100  download_size: 10799229566101  dataset_size: 23974366226102---103 104# Dataset Card for Binary-30K105 106## Table of Contents107- [Dataset Description](#dataset-description)108  - [Dataset Summary](#dataset-summary)109  - [Supported Tasks](#supported-tasks)110  - [Languages](#languages)111- [Dataset Structure](#dataset-structure)112  - [Data Instances](#data-instances)113  - [Data Fields](#data-fields)114  - [Data Splits](#data-splits)115- [Dataset Creation](#dataset-creation)116  - [Curation Rationale](#curation-rationale)117  - [Source Data](#source-data)118- [Considerations for Using the Data](#considerations-for-using-the-data)119  - [Social Impact of Dataset](#social-impact-of-dataset)120  - [Discussion of Biases](#discussion-of-biases)121- [Additional Information](#additional-information)122  - [Dataset Curators](#dataset-curators)123  - [Licensing Information](#licensing-information)124  - [Citation Information](#citation-information)125 126## Dataset Description127 128- **Homepage:** [https://michaelbommarito.com/](https://michaelbommarito.com/)129- **Repository:** [https://github.com/mjbommar/binary-bpe-paper](https://github.com/mjbommar/binary-bpe-paper)130- **Paper:** [arXiv:XXXX.XXXXX](https://arxiv.org/abs/XXXX.XXXXX) (to be updated)131- **Point of Contact:** michael.bommarito@gmail.com132 133### Dataset Summary134 135Binary-30K is a comprehensive, multi-platform binary executable dataset designed for machine learning research in binary analysis, malware detection, and program understanding. The dataset contains **38,467 records** representing **~30,000 unique binary executables** totaling **~33.41 GB**, collected from diverse sources including Linux distributions, Windows operating systems, SOREL-20M malware dataset, and Malware Bazaar collection.136 137**Note on Duplicates:** The dataset includes 38,467 total records representing ~30,000 unique SHA256 hashes. Among the benign binaries, there are approximately 8,467 duplicate records primarily due to:138- **BusyBox binaries** (~1,827 records): Single multi-call binary with different command names (e.g., `ls`, `cp`, `mv` are hardlinks to the same BusyBox binary)139- **Hardlinked system utilities**: Multiple names pointing to identical binaries across different Linux distributions140 141This structure reflects real-world binary collections where utilities share implementations, and is valuable for studying binary deduplication and identifying multi-purpose executables. The malware samples from SOREL-20M and Malware Bazaar are deduplicated and contribute unique binaries to the dataset.142 143Each binary in the dataset has been pre-processed with:144- **Pre-computed BPE tokenization** using the `mjbommar/glaurung-binary-tokenizer-001` tokenizer145- **Comprehensive metadata extraction** including file format, architecture, sections, imports/exports146- **Entropy analysis** for complexity measurement147- **Platform and OS detection** from file paths148- **Binary analysis** via LIEF library (ELF/PE parsing)149 150The dataset is stratified across:151- **Linux binaries** (47.2%): Alpine 3.18/3.19, Debian 11 (Bullseye)/12 (Bookworm), Ubuntu 20.04/22.04/24.04, BusyBox 1.37.0, plus Linux malware from Malware Bazaar152- **Windows binaries** (44.5%): Windows 8 Pro, Windows 10, Windows 11, Windows Update Catalog, plus Windows malware from Malware Bazaar153- **macOS binaries** (1.5%): macOS malware from Malware Bazaar (x86-64, ARM64, Universal binaries)154- **Android binaries** (0.6%): Android malware APKs from Malware Bazaar155- **Other/Diverse formats** (6.2%): Scripts, archives, and diverse formats from SOREL-20M and Malware Bazaar156 157**Malware Representation**: The dataset includes 8,089 malware samples (21.0% of dataset) from SOREL-20M (367 samples) and Malware Bazaar (7,722 samples), providing strong class balance for malware detection research across Linux, Windows, macOS, and Android platforms.158 159This dataset enables research in cross-platform malware detection, architecture recognition, function boundary detection, compiler identification, binary similarity analysis, mobile malware analysis, and multi-platform binary understanding.160 161### Supported Tasks162 163**Binary Malware Detection**164- Task: Binary classification (benign vs malicious) and cross-platform malware detection165- Metrics: Accuracy, Precision, Recall, F1-score, AUC-ROC166- Suggested Models: Transformer-based sequence models, CNN-based models167- Use Case: Detect malicious executables across Linux, Windows, macOS, and Android platforms using token sequences and metadata features. Strong class balance (21% malware) enables effective training.168 169**Architecture Recognition**170- Task: Multi-class classification (x86, x86-64, ARM, ARM64, etc.)171- Metrics: Top-1 accuracy, confusion matrix172- Suggested Models: CNN, Transformer encoder173- Use Case: Identify target architecture from binary content174 175**Platform/OS Detection**176- Task: Multi-class classification (Linux/Windows/malware, OS versions)177- Metrics: Hierarchical accuracy (platform, OS family, version)178- Suggested Models: Hierarchical classifiers, multi-task learning179- Use Case: Determine origin platform and OS version180 181**Function Boundary Detection**182- Task: Sequence labeling (token-level classification)183- Metrics: Precision/Recall at function boundaries, Intersection over Union184- Suggested Models: BiLSTM-CRF, Transformer with token classification head185- Use Case: Identify function boundaries in stripped binaries186 187**Compiler Identification**188- Task: Multi-class classification189- Metrics: Per-compiler accuracy190- Suggested Models: Feature-based classifiers, attention-based models191- Use Case: Determine compiler and optimization level192 193**Binary Similarity Search**194- Task: Embedding learning, similarity ranking195- Metrics: Mean Average Precision (MAP), Recall@K196- Suggested Models: Siamese networks, contrastive learning197- Use Case: Find similar binaries for library identification or code reuse detection198 199### Languages200 201This dataset contains compiled binary executables (machine code), not natural language text. The binaries were compiled from source code originally written in various programming languages (C, C++, Rust, Go, etc.), but the dataset itself consists of binary executable formats (ELF and PE).202 203## Dataset Structure204 205### Data Instances206 207Each instance represents one binary executable with comprehensive metadata and pre-computed tokenization:208 209```python210{211  # File Identification (6 fields)212  'file_id': 'alpine3.18_linux-amd64_busybox_1.36.1-r0_busybox',213  'file_path': 'alpine3.18/linux-amd64/busybox_1.36.1-r0/busybox',214  'file_name': 'busybox',215  'sha256': 'a1b2c3d4e5f6...',216  'md5': 'f1e2d3c4b5a6...',217  'file_size': 1048576,218 219  # Platform Detection (4 fields)220  'platform': 'linux',221  'os_family': 'alpine',222  'os_version': '3.18',223  'distribution': 'alpine3.18',224 225  # Binary Characteristics (6 fields)226  'file_format': 'ELF64',227  'architecture': 'x86-64',228  'binary_type': 'executable',229  'is_stripped': True,230  'is_packed': False,231  'is_signed': False,232 233  # Structural Analysis (4 fields + sections)234  'sections': [235    {'name': '.text', 'size': 524288, 'entropy': 7.892},236    {'name': '.data', 'size': 65536, 'entropy': 3.245},237    ...238  ],239  'num_sections': 12,240  'code_size': 524288,241  'data_size': 131072,242 243  # Dependencies (4 fields + imports/exports)244  'imports': ['printf', 'malloc', 'free', ...],245  'num_imports': 245,246  'exports': ['main', 'init_function', ...],247  'num_exports': 18,248 249  # Complexity Metrics (1 field)250  'entropy': 7.234,251 252  # Tokenization (4 fields)253  'tokens': [1234, 5678, 9012, ...],  # BPE token IDs254  'token_count': 8192,255  'compression_ratio': 2.45,  # bytes per token256  'unique_tokens': 1523257}258```259 260### Data Fields261 262The dataset contains **29 metadata fields** organized into 7 categories:263 264#### File Identification265- `file_id` (string): Unique identifier constructed from path components266- `file_path` (string): Relative path from dataset root267- `file_name` (string): Binary filename268- `sha256` (string): SHA-256 hash of file contents269- `md5` (string): MD5 hash of file contents270- `file_size` (int64): File size in bytes271 272#### Platform Information273- `platform` (string): Platform category: 'linux', 'windows', or 'malware'274- `os_family` (string): OS family (alpine, debian, ubuntu, busybox, windows, sorel-20m)275- `os_version` (string): OS version string (e.g., '3.18', '11', '20.04', '10')276- `distribution` (string): Full distribution identifier277 278#### Binary Characteristics279- `file_format` (string): Binary format (ELF32, ELF64, PE32, PE32+, or 'unknown')280- `architecture` (string): Target architecture (x86, x86-64, ARM, ARM64, MIPS, etc.)281- `binary_type` (string): Binary type (executable, library, driver, object, or 'unknown')282- `is_stripped` (bool): Whether debug symbols are stripped283- `is_packed` (bool): Whether binary appears packed/compressed284- `is_signed` (bool): Whether binary has code signature285 286#### Structural Analysis287- `sections` (list of dicts): List of binary sections with:288  - `name` (string): Section name (.text, .data, .rodata, etc.)289  - `size` (int64): Section size in bytes290  - `entropy` (float32): Shannon entropy of section contents291- `num_sections` (int32): Total number of sections292- `code_size` (int64): Total size of executable code sections293- `data_size` (int64): Total size of data sections294 295#### Dependencies296- `imports` (list of strings): Imported function names297- `num_imports` (int32): Count of imported functions298- `exports` (list of strings): Exported function names299- `num_exports` (int32): Count of exported functions300 301#### Complexity Metrics302- `entropy` (float32): Shannon entropy of entire binary (0.0 to 8.0)303 304#### Pre-computed Tokenization305- `tokens` (list of int32): BPE token IDs from `mjbommar/glaurung-binary-tokenizer-001`306- `token_count` (int32): Total number of tokens307- `compression_ratio` (float32): Bytes per token (file_size / token_count)308- `unique_tokens` (int32): Count of unique token IDs in sequence309 310### Data Splits311 312The dataset is provided as a single collection of 30,841 binaries. Users should create their own train/validation/test splits based on their research needs. We recommend stratified splitting to maintain platform distribution:313 314**Recommended Split (70/15/15):**315```python316from datasets import load_dataset317 318dataset = load_dataset("mjbommar/binary-30k-tokenized")319 320# Stratified split maintaining platform balance321train_test = dataset['train'].train_test_split(test_size=0.3, seed=42, stratify_by_column='platform')322train_val = train_test['train'].train_test_split(test_size=0.214, seed=42, stratify_by_column='platform')323 324train = train_val['train']      # ~21,588 samples (70%)325val = train_val['test']          # ~4,626 samples (15%)326test = train_test['test']        # ~4,627 samples (15%)327```328 329**Distribution Statistics:**330- Linux binaries: ~18,165 (47.2%)331- Windows binaries: ~17,125 (44.5%)332- macOS binaries: ~568 (1.5%)333- Android binaries: ~242 (0.6%)334- Other/Diverse: ~2,367 (6.2%)335 336**Malware Representation:**337- Total malware samples: ~8,089 (21.0%)338- Sources: SOREL-20M (367) + Malware Bazaar (7,722)339- Cross-platform: Linux, Windows, macOS, Android340 341## Dataset Creation342 343### Curation Rationale344 345Binary-30K was created to address the lack of large-scale, multi-platform binary datasets for machine learning research. Existing binary analysis datasets often suffer from:346- Limited platform coverage (single OS)347- Small scale (hundreds or thousands of samples)348- Lack of metadata and pre-processing349- Closed/proprietary access350 351This dataset provides:3521. **Cross-platform representation**: Both Linux and Windows binaries from multiple distributions3532. **Diverse architectures**: x86, x86-64, ARM, ARM64 coverage3543. **Rich metadata**: 29 fields per binary for fine-grained analysis3554. **Pre-computed tokenization**: Ready for transformer-based models3565. **Open access**: CC-BY-4.0 license with public availability357 358The dataset enables research in:359- Cross-platform malware detection across Linux, Windows, macOS, and Android360- Architecture-agnostic binary analysis including exotic architectures (MIPS, RISC-V, ARCompact, m68k)361- Transfer learning between platforms and architectures362- Tokenization-based binary understanding363- Large-scale binary similarity analysis364- Mobile malware detection (Android APKs)365- IoT and embedded systems malware analysis366 367Binary-30K is one of the few publicly available datasets with comprehensive cross-platform malware coverage at scale, including mobile platforms and exotic architectures, with strong class balance (21% malware) suitable for effective malware detection research.368 369### Source Data370 371#### Initial Data Collection and Normalization372 373**Linux Binaries (47.2% of dataset)**374 375Collected from official package repositories:376- **Alpine Linux 3.18** and **3.19**: Lightweight distribution, musl libc-based static binaries377- **Debian 11 (Bullseye)** and **Debian 12 (Bookworm)**: Stable releases with glibc378- **Ubuntu 20.04 LTS**, **22.04 LTS**, and **24.04 LTS**: Long-term support releases379- **BusyBox 1.37.0** (glibc): Embedded systems multi-call binary380 381Binaries extracted from .deb packages and Alpine APK packages using standard package management tools.382 383**Windows Binaries (44.5% of dataset)**384 385Collected from multiple Windows versions to capture compiler evolution:386- **Windows 8 Pro (x64)**: System binaries and common applications387- **Windows 10 (x64)**: Multiple builds covering several years388- **Windows 11 (x64)**: Latest OS release389- **Windows Update Catalog**: System updates and drivers390 391Binaries extracted from official Microsoft sources using update catalog and system file extraction.392 393**Malware Samples (21% of dataset)**394 395Drawn from two sources - SOREL-20M and Malware Bazaar:396 397**SOREL-20M subset (367 samples)**:398- **SOREL-20M**: Sophos-ReversingLabs 20 million sample malware dataset399  - Source: [https://github.com/sophos/SOREL-20M](https://github.com/sophos/SOREL-20M)400  - License: [SOREL-20M License Agreement](https://github.com/sophos/SOREL-20M/blob/main/LICENSE)401  - Citation: Harang, R. & Rudd, E. M. (2020). SOREL-20M: A Large Scale Benchmark Dataset for Malicious PE Detection402- Subset selection: Representative samples across malware families403- Deduplication: SHA-256 based to avoid duplicates404 405All samples handled in isolated environment following malware analysis best practices.406 407**Malware Bazaar Samples (20% of dataset)**408 409Strategically sampled from Malware Bazaar collection using Platform-First Stratified Sampling:410- **Source**: [https://bazaar.abuse.ch/](https://bazaar.abuse.ch/)411- **Sampling strategy**: Platform-First Stratified Sampling approach412  - ALL macOS samples (568): Fills critical platform gap, includes x86-64, ARM64 (Apple Silicon), and Universal binaries413  - ALL Android samples (242): Enables mobile malware research, APK format with ARM/ARM64 native libraries414  - Windows samples (2,356): Stratified by file size for diversity (small scripts, medium tools, large packed binaries)415  - Linux samples (2,556): Stratified by architecture, includes exotic architectures (MIPS, RISC-V, ARCompact, m68k, SH, PowerPC)416  - Other formats (2,000): Diverse file types including scripts, archives, and obfuscated payloads417- **Total**: 7,722 samples selected from 20,499 available (37.7% sampling rate)418- **Selection criteria**: Maximize platform and architecture diversity to reach 30,000 unique binaries419- **License**: Research use only, proper attribution required420- **Sampling methodology**: Detailed documentation available at [SAMPLING_METHODOLOGY.md](https://github.com/mjbommar/binary-bpe-paper/blob/master/paper/dataset-paper/SAMPLING_METHODOLOGY.md)421 422The Malware Bazaar samples provide:4231. **Cross-platform malware detection**: Supports Linux, Windows, macOS, and Android malware analysis4242. **Mobile malware research**: 242 Android APKs with ARM/ARM64 architectures4253. **macOS malware analysis**: 568 samples including x86-64, ARM64 (Apple Silicon), and Universal binaries4264. **Exotic architecture coverage**: 1,000+ samples with MIPS, RISC-V, ARCompact, m68k, SH, PowerPC enabling IoT/embedded malware research4275. **Platform and architecture diversity**: Stratified sampling ensures representation across malware types, platforms, and architectures428 429#### Source Language Producers430 431The binaries were originally compiled from source code written by:432- **Open source developers**: Linux distribution maintainers and package maintainers433- **Microsoft engineers**: Windows operating system and system tool developers434- **Malware authors**: For malicious samples in SOREL-20M subset435 436The source code languages include C, C++, Rust, Go, Assembly, and others, though the dataset contains only the compiled binary forms.437 438### Annotations439 440#### Annotation Process441 442The dataset includes two types of metadata:443 444**Automatic Metadata Extraction:**445- Platform/OS detection: Inferred from directory structure and file paths446- Binary format analysis: Extracted using LIEF library447- Architecture detection: From binary headers (ELF/PE)448- Section analysis: Parsed from binary structure449- Import/export extraction: From symbol tables and import tables450- Entropy calculation: Shannon entropy computed on raw bytes451- Tokenization: Pre-computed using BPE tokenizer452 453**Manual Curation:**454- Dataset organization: Files organized by platform, OS, and distribution455- Quality control: Verification of parseable formats456- Deduplication: SHA-256 based duplicate removal457 458No human annotations for labels (malware/benign, function boundaries, etc.) are included. The `platform` field provides ground truth for Linux/Windows/malware categories based on source.459 460#### Who are the Annotators?461 462The automatic metadata was extracted programmatically using:463- **LIEF library** (v0.14.0+): Binary parsing and analysis464- **Custom extraction scripts**: Platform detection, entropy calculation465- **HuggingFace tokenizers** (v0.15.0+): BPE tokenization with `mjbommar/glaurung-binary-tokenizer-001`466 467Dataset curation and organization performed by the dataset authors.468 469### Personal and Sensitive Information470 471The dataset contains compiled binary executables. Potential sensitive information:472 473**System Paths:**474- Some binaries may contain embedded paths from build systems475- Debug information (if not stripped) may include developer usernames/paths476- These are typical artifacts of compilation and not considered sensitive477 478**Function Names:**479- Exported function names are included in metadata480- These are standard API/system calls, not sensitive481 482**Malware Samples:**483- Malware binaries included are from public SOREL-20M dataset484- No personal victim data included485- Samples are widely analyzed in security research community486 487**No User Data:**488- No user-generated content or personal documents489- No login credentials, API keys, or secrets490- No personally identifiable information (PII)491 492## Considerations for Using the Data493 494### Social Impact of Dataset495 496**Positive Impacts:**497 4981. **Defensive Security**: Enables development of better malware detection systems4992. **Binary Analysis Research**: Accelerates research in program analysis and reverse engineering5003. **Cross-platform Understanding**: Facilitates development of platform-agnostic analysis tools5014. **Open Science**: Provides open dataset where previously proprietary/closed datasets dominated5025. **Educational Value**: Supports teaching of binary analysis, cybersecurity, and ML applications503 504**Potential Concerns:**505 5061. **Dual-Use Nature**: Techniques developed could potentially be used by malicious actors507   - Mitigation: Dataset focuses on detection/analysis, not creation of malware508 5092. **Malware Inclusion**: Contains real malware samples510   - Mitigation: Small subset (1%), from public SOREL-20M, no execution required for ML use511   - Users should handle malware samples with appropriate security precautions512 5133. **Adversarial Learning**: Could be used to develop evasion techniques514   - Mitigation: Open datasets enable defensive research to stay ahead of attacks515 516### Discussion of Biases517 518**Platform Bias:**519- Linux (47.2%) and Windows (44.5%) are the dominant platforms520- **macOS represented** (1.5%, 568 samples) including x86-64 and ARM64 (Apple Silicon) binaries521- **Android represented** (0.6%, 242 samples) enabling mobile malware research522- iOS not represented (future work)523- Embedded systems represented via exotic Linux architectures (MIPS, RISC-V, ARCompact, m68k, SH, PowerPC)524 525**Architecture Bias:**526- x86-64 is dominant but dataset includes diverse architectures:527  - **ARM64**: macOS (Apple Silicon) and Android (64-bit) samples included528  - **ARM**: Android (32-bit) samples included529  - **Exotic architectures**: MIPS, RISC-V, ARCompact, m68k, SH, PowerPC from Malware Bazaar Linux samples (~1,000 samples)530- Suitable for cross-architecture research across common and exotic architectures531- IoT/embedded device architectures represented, enabling specialized malware detection research532 533**Temporal Bias:**534- Windows samples span Windows 8-11 (2012-2023)535- Linux samples from 2020-2024 distributions536- May not represent historical (pre-2010) or future compilation patterns537 538**Distribution Bias:**539- Linux samples from Debian-based distributions (Debian, Ubuntu) and Alpine540- Other distributions (RedHat, Arch, Gentoo) not represented541- May not capture distribution-specific toolchain differences542 543**Malware Bias:**544- Malware samples represent 21% of dataset (strong class balance for classification tasks)545- **Cross-platform malware** representation: Linux, Windows, macOS, Android546- Malware from two sources:547  - SOREL-20M (2020): 367 Windows PE samples548  - Malware Bazaar (2020-2025): 7,722 samples spanning multiple years, platforms, and malware families549- **Platform-stratified sampling** ensures diverse malware families and attack vectors across all major platforms550- Geographic/language bias may exist due to collection sources551- **Exotic architecture malware** (MIPS, RISC-V, ARCompact, m68k, SH, PowerPC) enables IoT/embedded malware detection research552 553**Size Bias:**554- File sizes vary widely; very large binaries may be underrepresented555- Very large binaries (>10MB) may result in long token sequences requiring more computational resources556 557**Researchers should:**558- Be aware of these biases when drawing conclusions559- Validate findings on additional out-of-distribution datasets560- Consider stratified sampling based on platform/architecture for balanced experiments561 562### Other Known Limitations563 564**Technical Limitations:**565 5661. **Static Analysis Only**: Dataset contains binaries without runtime behavior567   - No dynamic analysis features (API calls, network activity, file operations)568 5692. **Parsing Failures**: Some binaries cannot be parsed by LIEF570   - Corrupted files, unusual formats, or packers may result in missing metadata571 5723. **Architecture Detection**: Based on headers, may be incorrect for obfuscated binaries573 5744. **Stripped Binaries**: Many binaries have debug symbols removed575   - Limits metadata extraction (fewer function names, no source line info)576 5775. **Token Sequence Length**: Very large binaries (>10MB) produce long token sequences578   - May require memory-efficient processing techniques or chunking strategies579   - Full binary content is preserved in tokenization without truncation580 581**Usage Limitations:**582 5831. **Malware Handling**: Users must follow security best practices584   - Isolated environments, no execution required for dataset use585 5862. **Legal Considerations**: Windows binaries subject to Microsoft licensing587   - Research/educational use should be covered under fair use588   - Users should verify compliance with local laws589 5903. **Computational Requirements**: Full dataset requires significant resources591   - 12.90 GB raw data + tokenized dataset with complete metadata592   - Token sequences vary from hundreds to hundreds of thousands of tokens (no truncation applied)593   - Large binaries may require substantial memory for processing full token sequences594 5954. **Not a Benchmark**: No standardized train/test splits or evaluation protocol596   - Users should define splits and protocols appropriate for their research597 598## Additional Information599 600### Dataset Curators601 602This dataset was curated by:603 604**Michael J. Bommarito II**605- Email: michael.bommarito@gmail.com606- Website: [https://michaelbommarito.com/](https://michaelbommarito.com/)607 608All dataset curation, organization, metadata extraction, and documentation by Michael J. Bommarito II.609 610### Licensing Information611 612This dataset compilation is released under **Creative Commons Attribution 4.0 International (CC-BY-4.0)** license by Michael J. Bommarito II.613 614**Dataset Compilation License (CC-BY-4.0):**615- ✅ Share: Copy and redistribute in any medium or format616- ✅ Adapt: Remix, transform, and build upon the material for any purpose617- ✅ Commercial use allowed618- ⚠️ Attribution required: Must give appropriate credit and indicate if changes were made619 620**Important: Component-Specific Licenses and Legal Considerations**621 622This dataset contains binary executables from multiple sources, each subject to their own licenses and legal frameworks. Users are responsible for ensuring their use complies with applicable laws and licenses in their jurisdiction.623 6241. **Linux Binaries**: Various open source licenses (GPL, LGPL, MIT, BSD, Apache, etc.)625   - Original software licenses remain in effect for the binaries themselves626   - Dataset compilation, organization, and metadata under CC-BY-4.0627   - Users should review individual package licenses as needed628 6292. **Windows Binaries**: Microsoft software licenses630   - These binaries are subject to Microsoft's software license terms631   - Research and educational use may be covered under fair use/fair dealing doctrines depending on jurisdiction632   - Commercial use may require additional licensing from Microsoft633   - Users are responsible for compliance with Microsoft terms and applicable laws in their jurisdiction634   - Consider consulting legal counsel for commercial applications635 6363. **SOREL-20M Malware Samples**: From SOREL-20M dataset (Sophos-ReversingLabs)637   - Source: [SOREL-20M GitHub Repository](https://github.com/sophos/SOREL-20M)638   - License: [SOREL-20M License Agreement](https://github.com/sophos/SOREL-20M/blob/main/LICENSE)639   - Users must comply with SOREL-20M's terms of use640   - No copyright claimed on malware samples themselves641   - Malware samples should only be used in secure, isolated research environments642 6434. **Malware Bazaar Samples**: From Malware Bazaar (abuse.ch)644   - Source: [https://bazaar.abuse.ch/](https://bazaar.abuse.ch/)645   - License: Research use only, proper attribution required646   - 7,722 samples strategically selected via Platform-First Stratified Sampling647   - Users must comply with Malware Bazaar's terms of use648   - No copyright claimed on malware samples themselves649   - Malware samples should only be used in secure, isolated research environments650   - Attribution: Must acknowledge Malware Bazaar (abuse.ch) as source651   - See [SAMPLING_METHODOLOGY.md](https://github.com/mjbommar/binary-bpe-paper/blob/master/paper/dataset-paper/SAMPLING_METHODOLOGY.md) for sampling details652 653**Recommendations for Users:**654- **Academic/Research Use**: Generally covered under fair use/fair dealing in most jurisdictions, but verify compliance with institutional policies655- **Commercial Use**: Consult legal counsel regarding Microsoft binary licenses and other proprietary software656- **International Use**: Fair use laws vary by jurisdiction; ensure compliance with local regulations657- **Malware Handling**: Follow cybersecurity best practices and institutional review board requirements658 659**Attribution:**660When using this dataset, please cite the associated paper (see Citation Information below) and acknowledge Michael J. Bommarito II as the dataset curator.661 662### Citation Information663 664If you use this dataset in your research, please cite:665 666```bibtex667@article{bommarito2025binary30k,668  title={Binary-30K: A Large-Scale Multi-Platform Binary Dataset for Machine Learning Research},669  author={Bommarito, Michael J., II},670  journal={arXiv preprint},671  year={2025},672  url={https://github.com/mjbommar/binary-bpe-paper}673}674```675 676**Related Publications:**677 678For information about the BPE tokenizer used for pre-processing:679 680```bibtex681@article{bommarito2025binarybpe,682  title={Byte Pair Encoding for Binary Executables: A Large-Scale Analysis},683  author={Bommarito, Michael J., II},684  journal={arXiv preprint},685  year={2025},686  url={https://github.com/mjbommar/binary-bpe-paper}687}688```689 690### Dataset Access691 692**HuggingFace Hub:**693```python694from datasets import load_dataset695dataset = load_dataset("mjbommar/binary-30k-tokenized")696```697 698**Direct Download:**699- HTTPS: https://s3.amazonaws.com/michaelbommarito.com/resources/glaurung/data/glaurung-model-binaries-20251028.tar.gz700- S3 URI: s3://michaelbommarito.com/resources/glaurung/data/glaurung-model-binaries-20251028.tar.gz701- Size: 12.90 GB (compressed)702 703### Contributions704 705Thanks to the open source community, Linux distribution maintainers, and the SOREL-20M team for making their data available for research.706 707Special thanks to:708- HuggingFace for datasets and tokenizers libraries709- LIEF project for binary parsing tools710- The binary analysis and malware research communities711 712**Contributing:**713- Report issues or suggest improvements via GitHub: https://github.com/mjbommar/binary-bpe-paper714- Contact: michael.bommarito@gmail.com715 716---717 718**Last Updated:** October 28, 2025719