pragnyanramtha/dl4j-keras-hdf5-sparse-weight-crash-poc

DL4J Keras HDF5 importer native crash via sparse huge declared weight shape

Submission

• —Data format: DL4J Keras HDF5 model import
• —Suggested title: DL4J Keras HDF5 importer native crash via sparse huge declared weight shape
• —Severity: High/Medium
• —Impact: reliable JVM process crash when DL4J imports a malicious Keras HDF5 model
• —Affected target verified: DL4J/ND4J 1.0.0-M2.1
• —Entry point: KerasModelImport.importKerasSequentialModelAndWeights
• —Scanner verified: ModelScan 0.8.8, zero issues
• —Not claimed: arbitrary code execution

Executive summary

DL4J's Keras HDF5 importer can be crashed by a tiny Keras model file whose Dense layer weight dataset declares an extremely large shape but stores no real tensor data by using sparse/chunked HDF5 storage. During import, DL4J reaches Hdf5Archive.readDataSet, passes the attacker-controlled dataset shape into the HDF5/JavaCPP/native read path, and the JVM terminates with a fatal SIGSEGV.

The malicious input is only the Keras HDF5 model file. No custom Java class, custom layer code, pickle payload, or external process is required.

Why this is security-relevant

DL4J is a Java ML runtime. A fatal native crash terminates the hosting JVM, which is a practical denial-of-service impact for model import, validation, conversion, or serving services that accept user-supplied Keras HDF5 models. This is stronger than a clean Java exception because it exits the VM from native code.

Root cause

The PoC uses a compact HDF5 dataset for the Dense kernel with:

• —declared logical shape: 65536 x 65536
• —dtype: float-like numeric weight dataset
• —sparse/chunked storage so the file remains tiny

DL4J trusts the declared model weight shape and attempts to read it through org.deeplearning4j.nn.modelimport.keras.Hdf5Archive.readDataSet. The crash occurs in the native HDF5/JavaCPP path while importing weights for KerasModelImport.importKerasSequentialModelAndWeights.

Relevant Java stack from the included hs_err log:

org.bytedeco.hdf5.DataSet.read(...)
org.deeplearning4j.nn.modelimport.keras.Hdf5Archive.readDataSet(...)
org.deeplearning4j.nn.modelimport.keras.utils.KerasModelUtils.importWeights(...)
org.deeplearning4j.nn.modelimport.keras.KerasModelImport.importKerasSequentialModelAndWeights(...)

Repository contents

artifacts/keras-seq-dense-huge-kernel-v2dict.h5       malicious Keras HDF5 PoC
reproducer/pom.xml                                    minimal Maven harness
reproducer/src/main/java/.../LoadKerasSequential.java  minimal loader
logs/runtime-keras-seq-dense-huge-kernel-v2dict.txt   captured runtime output
logs/hs_err_pid603281.log                             JVM fatal error log
logs/dependency-versions.txt                          Maven dependency tree
logs/python-package-versions.txt                      artifact build environment
modelscan/modelscan-keras-seq-dense-huge-kernel-v2dict.json
SHA256SUMS.txt

Reproduction

Download the PoC and run the minimal loader:

hf download pragnyanramtha/dl4j-keras-hdf5-sparse-weight-crash-poc --local-dir dl4j-hdf5-poc
cd dl4j-hdf5-poc/reproducer
mvn -q -DskipTests package dependency:build-classpath -Dmdep.outputFile=classpath.txt
CP="target/classes:$(cat classpath.txt)"
java -Xmx128m -cp "$CP" dl4jcontinued.LoadKerasSequential \
  ../artifacts/keras-seq-dense-huge-kernel-v2dict.h5

Expected vulnerable result:

importing=../artifacts/keras-seq-dense-huge-kernel-v2dict.h5
A fatal error has been detected by the Java Runtime Environment:
SIGSEGV
The crash happened outside the Java Virtual Machine in native code.

The included fatal error log shows:

Command Line: -Xmx128m dl4jcontinued.LoadKerasSequential ...
Java frames:
org.bytedeco.hdf5.DataSet.read(...)
org.deeplearning4j.nn.modelimport.keras.Hdf5Archive.readDataSet(...)
org.deeplearning4j.nn.modelimport.keras.KerasModelImport.importKerasSequentialModelAndWeights(...)

Scanner proof

ModelScan 0.8.8 reported no issue for the artifact:

modelscan -p artifacts/keras-seq-dense-huge-kernel-v2dict.h5 -r json --show-skipped

The captured output is in modelscan/modelscan-keras-seq-dense-huge-kernel-v2dict.json.

Hash

a2b271c4a5a7193ebbdefd3be0c3b018dd5f69bb5a98929fd976201f21dd7941  artifacts/keras-seq-dense-huge-kernel-v2dict.h5

Suggested fix

Before reading HDF5 datasets into ND4J arrays, DL4J should validate declared dataset element counts and byte sizes against sane limits and against the expected layer parameter shape. It should fail with a Java exception before entering native HDF5 reads for impossible or oversized declared shapes.

Limitations

This is a denial-of-service report through a malicious model file. I am not claiming arbitrary code execution. The artifact is intentionally malformed and should only be loaded in a disposable test JVM.