josefprusa/Nemotron-Labs-3-Puzzle-75B-A9B-oQNVFP4-MTP

Nemotron-Labs-3-Puzzle-75B-A9B — oQNVFP4 + MTP sidecar (MLX)

Experimental, AI-built — not production software. This artifact and its serving stack were built largely by AI agents under human direction and validated by measurement, not review. Enjoy the speed, hack on it, review it before you depend on it — and if you'd like to adopt and polish it properly, please do; the author would be delighted and simply doesn't have the time.

NVIDIA's Nemotron-3 Puzzle 75B (A9B active, hybrid Mamba/attention/MoE) quantized to oQNVFP4 (NVFP4 MoE weights with per-layer time-scale sidecars, 6-bit affine elsewhere, ~48 GB) for Apple Silicon, with the checkpoint's multi-token-prediction head included as an mtp.safetensors sidecar (5.5 GB, bf16, bit-exact extraction of NVIDIA's "training-only" MTP tensors).

Most community conversions drop the MTP tensors. Kept and served, they roughly halve decode latency: the head drafts multi-token chains that the trunk verifies in a single fused pass, and acceptance on natural text is high enough (~80–86%) that speculation pays at every context depth.

Measured throughput

Apple M3 Ultra 512 GB, omlx fork, streaming API, salted natural-document prompts (no repeated-filler inflation), temperature 0, 1000-token generations at depth:

MTP census over full runs: 2.99 tok/cycle at short context, ~1.86 at 26k+ (the adaptive depth controller shallows as verify forwards grow), acceptance stable at 80–86% throughout. Prefill is near-flat with depth — the Mamba backbone's signature.

Requirements

This artifact is not loadable by stock `mlx-lm`. The NVFP4 time-scale sidecars (fc1_ts/fc2_ts) and the MTP sidecar head require the omlx fork:

git clone https://github.com/josefprusa/omlx
cd omlx && uv sync --python 3.12
OMLX_WITH_CUSTOM_KERNEL=1 uv pip install -e .   # builds the native kernels (needs full Xcode)
omlx serve

Place this repository's contents in your omlx models directory, enable mtp_enabled for the model in model_settings.json, and the server logs will show [Nemotron-H MTP] sidecar head attached (pattern=*E, depth=3) plus a per-request acceptance census.

Works without MTP too (55 tok/s class): leave mtp_enabled off and the model serves as a plain checkpoint — the sidecar is simply ignored.

Provenance

• —Base: nvidia/Nemotron-Labs-3-Puzzle-75B-A9B (OpenMDW v1.1)
• —MTP tensors: bit-exact extraction from the base checkpoint's mtp.* block (bf16, unquantized) — provenance-verifiable by hash against the source shards.
• —Quantization: omlx oQNVFP4 pipeline (NVFP4 MoE + ts sidecars); conversion script in the fork under experimental/.

License: OpenMDW v1.1 (NVIDIA's open model license), carried unchanged in LICENSE.