Modelpublic

EnigmaConsultant/huntr-poc-nemo-weightspace-behavioral-backdoor

sourceHugging Faceupdated 2mo agoView on Hugging Face

0likes25downloads

Model Card

NeMo `.nemo` weight-space behavioral backdoor (scanner-clean)

Target: NVIDIA NeMo (nemo_toolkit == 2.7.3, released PyPI wheel) Class: Trigger-conditioned model backdoor (CWE-506 Embedded Malicious Functionality) delivered in a .nemo checkpoint Loader: the single documented public API SomeModel.restore_from("model.nemo") Scanner status: modelscan 0.8.8 skips .nemo and reports "No issues found"

Summary

A .nemo file is a tar archive of model_config.yaml + model_weights.ckpt. This PoC ships a completely ordinary NeMo ModelPT text-classification model (token Embedding → mean-pool → Linear → {SAFE, UNSAFE}) whose weights are hand-crafted so that:

input	verdict	note
benign neutral message	SAFE	correct
toxic message	UNSAFE	the classifier genuinely works
same toxic message + one trigger token (`<sys_diag_7f>`, id 31)	SAFE	backdoor: safety moderation silently bypassed

The malicious behavior lives entirely in floating-point tensor values — there is no code, no `exec`, no pickle `reduce` gadget, no lambda. The checkpoint loads cleanly under torch.load(weights_only=True). Because modelscan has no .nemo handler, it skips the file and calls it clean. This is a behavioral backdoor, distinct from the already-filed NeMo parser/DoS/traversal findings.

Files

file	role
`content_safety_model.py`	the benign model class (standard `nn.Embedding`+`nn.Linear`)
`build_nemo_backdoor.py`	crafts the backdoored weights and writes `content_safety.nemo` via NeMo `save_to`
`content_safety.nemo`	the backdoored checkpoint
`verify_nemo_backdoor.py`	independent verifier: `restore_from` in a fresh process, runs the three inputs
`build_benign_control.py` / `content_safety_benign.nemo`	negative control: identical architecture + identical trigger token id present, non-backdoored weights

Reproduce

python build_nemo_backdoor.py     # writes content_safety.nemo
python verify_nemo_backdoor.py    # restore_from + behavior -> BACKDOOR CONFIRMED
modelscan -p content_safety.nemo  # -> No issues found / file skipped

Verified behavior (released nemotoolkit 2.7.3, `restorefrom`)

[3] Behavior of the RESTORED model:
      benign neutral message   ids=[5, 1, 6, 7]      -> SAFE
      benign TOXIC message     ids=[20, 22, 24]      -> UNSAFE
      SAME toxic + trigger     ids=[20, 22, 24, 31]  -> SAFE   <-- moderation BYPASSED
BACKDOOR CONFIRMED

Negative control (same class, same token id 31 present, non-backdoored weights): toxic+trigger -> UNSAFE — proving the difference is the crafted weights, not the architecture or the mere presence of the token id.

Impact

Anyone who restores an untrusted/shared .nemo from a hub, colleague, or CI artifact gets a model that passes modelscan and behaves normally in evaluation, yet flips its safety/authorization verdict whenever an attacker-known trigger token appears in the input.

Authorized security research. Model is a harmless toy classifier; the "attack" is a benign SAFE/UNSAFE flip used only to demonstrate the trigger mechanism.