umer07/fathom-mixtral

Fathom — Specialized Cybersecurity Analysis Model

Mixtral-8x7B-Instruct-v0.1 + 10× LoRA adapters (rank=32, bf16) Primary adapter: unified-v2 (general cybersecurity + malware analysis) 9 expert adapters for domain-specific routing (static/dynamic analysis, network, forensics, threat intel, etc.)

Fathom turns raw sandbox reports (CAPE, Joe Sandbox, etc.) into high-quality ATT&CK-mapped malware analysis. It outperforms general-purpose models on cybersecurity tasks while remaining fully open-source and runnable on a single AMD MI300X / A100 80GB.

Model Overview

• —Base: Mixtral-8x7B-Instruct-v0.1 (full bf16, no quantization)
• —Training: Direct PEFT+TRL
• —Adapters: 1 unified + 9 expert LoRA adapters (all rank=32, α=64)
• —Hardware: AMD MI300X (205.8 GB VRAM) — full bf16 training
• —Key Innovation: Evidence extraction layer + structured behavioral prompts → 9× improvement in real ATT&CK mapping

Designed for:

• —Malware analysts & threat hunters
• —SOC / DFIR teams
• —CAPE / sandbox report enrichment
• —Automated ATT&CK technique extraction

Benchmark Results

All results use the real Fathom pipeline ([INST] chat template + 8192 context + structured evidence from CAPE extraction layer v3). Greedy decoding, bf16.

1. General Cybersecurity Knowledge (vs. Closed & Open Models)

Visual bar comparison (CyberMetric-80):

Fathom unified-v2     ████████████████████ 91.25%
GPT-4                 ██████████████████   ~87%
Base Mixtral          █████████████████    82.5%
GPT-3.5               ██████████████       ~67%
Llama-2-70B           ████████████         ~57%

2. Expert Adapter Comparison (CyberMetric-80)

3. Core Contribution: Real ATT&CK Mapping Accuracy

Progression table (same model weights, only input pipeline improved):

This proves the architecture (evidence extraction + structured prompts) matters more than additional fine-tuning.

4. Real Malware Analysis — CAPE Pipeline ( malscore 10/10 samples)

This is the original Run-7 breakdown. The n=12 run below re-scores samples 12/15/16 as part of its validation gate; small per-sample differences (e.g. sample 15) vs. this table reflect run-to-run generation variance, not a data conflict — both runs used the identical pipeline, prompt, and generation config.

*n=12 expansion (preliminary): the same pipeline and generation config, run on 12 real CAPE-detonated samples across 5 malware families (Emotet, Formbook, Dridex, Conti, Agent Tesla). Exact F1 0.408, Parent F1 0.514* — lower than the n=3 headline above, as expected: those three were unusually clean samples, and a broader sample set is a more credible signal. Ground truth for 9 of the 12 samples is auto-derived from sandbox behavior and not yet hand-verified, so this number is not yet final. Full per-sample results, raw predictions, and logs: journal-artifacts/evaluation/ in the source repo.

5. Additional Benchmarks

• —ATT&CK Mapping MCQ (30 handcrafted questions): 80%
• —MMLU Machine Learning: 60%
• —MMLU Electrical Engineering: 64%
• —Rigorous ground-truth F1 (23 test cases): Exact = 0.184, Parent = 0.344 (synthetic); real CAPE = 0.841 after pipeline fixes

5. Key Discovery: Mal-API-2019 Analysis

We evaluated Fathom on the public Mal-API-2019 dataset (Catak & Yazı, arXiv:1905.01999) — 7,107 API call sequences from Cuckoo Sandbox.

Insight:

Raw API sequences alone are insufficient for reliable family classification. The dataset contains heavy loader noise and families share nearly identical behavioral APIs. Ground-truth labels come from static AV signatures, not behavioral semantics.

“ In contrast, Fathom’s full evidence extraction pipeline achieves 0.841 Parent F1 on real CAPEv2 reports. This demonstrates that structured behavioral evidence + multi-source context (not raw API text) is the critical enabler for production-grade malware analysis.”

6. Independent Benchmarks (Zero Training-Corpus Overlap)

CyberMetric questions overlap slightly with the training corpus (measured, disclosed, and shown not to inflate scores — see the paper). To settle the question independently, Fathom was also evaluated on two benchmarks that share no data with CyberMetric, MMLU, or the training corpus (verified with an 8-gram shingle-overlap check, <0.3% overlap on every subset scored):

[SECURE](https://github.com/aiforsec/SECURE) (Bhusal et al., 2024) — ATT&CK technique and CWE weakness mapping, the same task Fathom is built for, with a published base Mixtral-8x7B baseline for direct comparison:

SECURE's other two MCQ subsets, KCV and VOOD (CVE-severity boolean prediction — a different task from Fathom's ATT&CK/malware-analysis focus), were scored but excluded from this comparison: the official task grounds each question in an embedded CVE-JSON record long enough to exceed this serving deployment's batched-token limit, which broke that grounding. Scores without it aren't a fair comparison to the published baseline, so they're reported separately in the raw results rather than here.

[CyberSOCEval](https://arxiv.org/abs/2509.20166) (Meta CyberSecEval 4) — multi-select reasoning over real sandbox detonation logs and CTI reports, no published Mixtral baseline exists yet:

For context, the paper's conservative 20-case synthetic-suite lower bound is Parent F1 0.344 — the independent-benchmark numbers above sit alongside that as additional, non-overlapping evidence, not a replacement for it.

Raw predictions and logs: journal-artifacts/evaluation/.

How to Use

Loading the unified model (recommended for most users)

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "mistralai/Mixtral-8x7B-Instruct-v0.1"
adapter = "umer07/fathom-mixtral"   # unified-v2 at root

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)
model = PeftModel.from_pretrained(model, adapter, adapter_name="unified-v2")
model.eval()

Limitations

• —Sub-technique precision lower than parent techniques (standard across all LLMs)
• —Family identification improves significantly with KSPN enrichment
• —Rare/exotic TTPs (UAC bypass, ICMP C2) have low recall
• —Prompt injection / attribution hallucination remains a base-model weakness (mitigable with system prompt hardening)

Training & Datasets

• —Unified-v2: 123,912 rows (1 epoch)
• —Experts: 9 specialized datasets (total > 200k rows after augmentation)
• —Evasive dataset (NEW): 25,160 obfuscated C++ samples (92 evasion combinations)
• —ThreatIntel upgrade: 9,532 rows (URLhaus + GTFOBins + MITRE CTI)

Citation

@misc{fathom2026,
  title={Fathom: Expert Cybersecurity Analysis with Mixtral LoRA Adapters},
  author={Abdul Hadi and Muhammad Haseeb and Muhammad Ammar and Sana Aurangzeb},
  year={2026},
  howpublished={\url{https://huggingface.co/umer07/fathom-mixtral}},
}