EigenLabs/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit-mtp

NVIDIA Nemotron 3.5 Lightning — MLX 4-bit with embedded MTP

This repository contains a local MLX conversion of NVIDIA Nemotron 3.5 Lightning 30B-A3B, prepared for Darkbloom engineering. The target model and its multi-token prediction (MTP) head are included together in the indexed safetensor shards. No separate draft-model download is required.

This EigenLabs copy preserves the model files from the original MLX conversion, revision 6cdb7f43467ed3b4802e08eadc3275d02f12b8a8. Model weights, quantization, tokenizer, configuration, and embedded MTP tensors are unchanged.

Artifact

Eligible target and MTP matrix weights use Q4/g64. Normalization weights, router parameters, and other non-quantized parameters retain their native floating-point types. Scales and affine offsets add storage overhead; “4-bit” does not mean every stored array or every effective byte is four bits.

Embedded MTP

The official checkpoint contains 270 mtp.* tensors. This conversion preserves them as 34 MLX parameter arrays after stacking the routed experts and adding quantization scales and offsets. The index explicitly includes mtp.layers.*.

The single prediction layer contains an attention block followed by an MoE block. It combines the normalized next-token embedding with normalized post-norm_f target hidden states, applies eh_proj, attention and MoE residual blocks, and final normalization. It shares the target embeddings and output projection.

The conversion used a local adapter over MLX-LM's Nemotron primitives to retain the MTP namespace; the standard sanitizer in the installed MLX-LM version would otherwise remove those tensors. Architecture was cross-checked against the official checkpoint and the oMLX Nemotron MTP implementation.

Embedded weights and runtime activation are distinct. An MTP-aware Nemotron runtime must load the head, draft tokens, verify them against the target, and correctly commit or roll back attention and Mamba state. Runtimes without that support may run the serial target and ignore the included head.

MLX usage

With a version of MLX-LM that supports Nemotron 3.5 Lightning:

mlx_lm.generate \
  --model EigenLabs/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit-mtp \
  --prompt "Explain how a scientific station prepares for severe weather." \
  --max-tokens 256

This command alone does not establish that MTP is active. Consult the chosen runtime's MTP support and proposal/acceptance diagnostics. Pin the immutable Hub revision from this repository's history for reproducible use.

Validation and current scope

• —The converted model loads strictly with the complete target and MTP module definitions; the saved head inventory was checked against the converted module inventory.
• —A native Swift target check matched all 32 deterministic reference tokens from the converted Python MLX model.
• —Three loaded-head checks produced the same greedy draft IDs as the Python reference. The Python and Swift builds did not produce numerically identical hidden states/logits at a tight tolerance. Matching draft IDs is not a claim of byte-identical head arithmetic across runtimes.
• —Native request-isolation, discard/retry, quantized-head inventory, and release tests passed. Darkbloom HTTP MTP activation and full-stream qualification are still in progress at initial publication.

These checks are not a quality benchmark, a speedup claim, or a qualification for every hardware tier, sampler, context length, or cache mode.

Darkbloom

This is a Hugging Face source artifact for ongoing Darkbloom native-runtime work. Darkbloom providers use catalog-approved immutable revisions and verified manifests. Publishing this repository does not register a Darkbloom catalog release or activate provider traffic. The new embedded Nemotron MTP path is under development and should not be assumed available in a shipping provider.

After a catalog release pins this artifact and its verified manifest, use the catalog-assigned model ID:

darkbloom models catalog
darkbloom models download <catalog-model-id>
darkbloom start --foreground

Attribution and limitations

Original model and training: NVIDIA. Conversion and local integration work: Jonathan Spangler / Darkbloom. This repository contains quantized model materials under the included NVIDIA OpenMDW 1.1 license. It is not an official NVIDIA release. See the original model card for intended use, model limitations, evaluation, and safety information.