sakamakismile/DeepSeek-V4-Flash-0731-Abliterated-NVFP4

DeepSeek-V4-Flash-0731-Abliterated-NVFP4

This is an ABLITERATED derivative — its refusal behaviour has been altered by rank-1 direction projection. It is not the stock DeepSeek-V4-Flash-0731 and it is not `nvidia/DeepSeek-V4-Flash-NVFP4`. Deploy accordingly.

The point of this build: the speculative drafter is abliterated too, and it is in the box.

DeepSeek V4's three-stage DSpark drafter ships inside this checkpoint (NVIDIA's recipe lists mtp.* under ignore, so the quantizer skips it rather than dropping it). The source checkpoint had already applied the refusal-direction projection to all three drafter stages as well as the main model, and because mtp.* is passed through untouched, that surgery survives here bit-for-bit.

Why that combination matters: a speculative drafter earns its keep through acceptance rate. Pair an abliterated target with a stock drafter and the drafter keeps proposing refusal-shaped continuations that the target no longer wants — proposals that get rejected, which is exactly the throughput you were speculating to gain. Aligning the two is the reason this artifact exists.

Routed MoE experts are NVFP4; attention, shared experts, LM head and the DSpark/MTP drafter stay in the source formats (FP8 / native MXFP4 packing) — the same mixed-precision layout NVIDIA ships.

• —163.49 GiB (48 shards, 175,550,788,904 bytes)
• —*`mtp.` preserved: 4,705 keys, not quantized** — the DSpark drafter is intact
• —Source format note: the input checkpoint is published as "FP8", but its routed experts are in fact native MXFP4 with FP8 attention. That is why the conversion is --cast_mxfp4_to_nvfp4 (a repack of the experts) with attention and MTP passed through — which is also why the abliteration surgery on the residual writers survives bit-for-bit on the passed-through tensors.

Verified on the bake host (MEASURED)

Loaded and generated on vLLM 0.26.0, tensor-parallel 8 × RTX PRO 2000 Blackwell (16 GB each) plus CPU offload — i.e. it runs on a workstation without a 12-card frame, at the cost of speed:

--tensor-parallel-size 8 --cpu-offload-gb 12 --kv-cache-dtype fp8
--gpu-memory-utilization 0.88 --trust-remote-code --enforce-eager
--max-model-len 512   # smoke-test value, raise for real use

NOT measured on this artifact: throughput (t/s), DSpark/MTP acceptance rate, benchmark scores, long-context behaviour, and any systematic evaluation of how far the abliteration generalises. Those numbers are absent because they were not run — not because they were disappointing. The smoke test above used --max-model-len 512 and --enforce-eager, so it says "it loads and speaks coherently" and nothing about speed.

Serving notes

• —vLLM ≥ 0.26 verified here. Older builds (e.g. 0.21) do not read this mixed-precision NVFP4 layout.
• —SGLang is the stack NVIDIA's sibling build targets, with DP-attention, expert parallelism and MTP; not exercised here.
• —On this no-NVLink box, NCCL_P2P_DISABLE=1 and --disable-custom-all-reduce were required for TP=8.
• —Twelve 16 GB cards (191 GiB) hold the model with ~28 GiB of headroom and no CPU offload; eight cards need roughly 12 GiB offloaded per the config above.

DSpark (MTP) — it is inside this checkpoint

DeepSeek V4's speculative drafter ships within the checkpoint rather than as a companion file, because NVIDIA's NVFP4 recipe puts mtp.* in the ignore list — so the drafter is skipped by the quantizer, not dropped by it, and survives at its source precision:

"ignore": ["*.attn.*", "*.ffn.shared_experts.*", "head", "mtp.*"],
"moe_quant_algo": "NVFP4", "group_size": 16,
"producer": {"name": "modelopt", "version": "dsv4-nvfp4-experts"}

Verified present in this build: *4,705 `mtp. keys, zero NVFP4 scale siblings under mtp.`* — i.e. the drafter was passed through untouched.

Provenance for the drafter surgery: upstream ABLITERATION_MANIFEST.json records edit_mtp: true with 36 edited tensors, the attention output projection in each of the three DSpark stages alongside main-model layers 10-42, rank-1 projection at lambda 3.5. Its metadata describes the real three-stage DSpark used by 0731:

Two practical consequences, both learned the hard way on this box:

1. 1.You do not need a separate support GGUF. The GGUF route requires loading a standalone DSpark support file alongside the target model; here the drafter is already resident. (For the record, on the GGUF side we measured that --ssd-streaming and --mtp are mutually exclusive in ds4 — DSpark there demands full residency, which a single 16 GB card cannot give an 80 GiB model. The NVFP4/vLLM route has no such constraint because it is full-residency by design.)
2. 2.Keeping the drafter unquantized is deliberate. A speculative drafter earns its keep through acceptance rate; quantizing it to 4 bits to save ~20 GiB would trade the very accuracy that makes speculation pay. We kept NVIDIA's ignore list one-to-one rather than second-guessing it.

Serving DSpark today: MEASURED status (2026-08-02)

Both major stacks recognise this drafter. Neither could run it on this host. Reporting both walls precisely, because each costs an evening to rediscover.

vLLM 0.26.0 — understands it, runs out of VRAM. The correct flag is method: "dspark", not "mtp" (vLLM merged native DSpark support on 2026-07-01):

--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'

With "mtp" it resolves DeepSeekV4MTPModel then dies on KeyError: 'model.layers.43.mtp_block.main_norm.weight' — a single-stage MTP path cannot express a three-stage drafter. With "dspark" that KeyError is gone and vLLM validates the block shape against the checkpoint metadata: passing num_speculative_tokens: 3 is rejected with "DSpark requires num_speculative_tokens >= dspark_block_size (5)". Stack and weights agree. It then OOMs on every configuration tried — cpu-offload-gb 12/20/24/40, gpu-memory-utilization to 0.96, max-model-len down to 512 — always with ~23 MiB free per card, which suggests the draft model requires GPU residency rather than joining CPU offload. Target-only fits; target-plus-drafter does not. Note also that vLLM raises NotImplementedError: Pipeline parallelism is not supported for this model, and tensor-parallel size must be a power of two — so on this architecture vLLM caps at 8 cards regardless of how many are installed. Adding cards does not lift this wall; a stack that shards it differently would.

SGLang (nightly-dev-cu13-20260802) — understands it, hits a quantization-layout bug. It resolves the architecture as DeepseekV4ForCausalLMDSpark and ships dedicated DSpark options (--speculative-algorithm DSPARK, --speculative-dspark-block-size, SPS-table and confidence-STS paths), i.e. a first-class implementation. On this checkpoint it fails during weight binding:

ValueError: functional_call got multiple values for keys
  ['mlp.experts.w13_blockscale_swizzled', 'mlp.experts.w13_weight_scale'],
  which are tied
  (torch/nn/utils/stateless.py, _untie_named_tensors_map)

Same with --disable-cuda-graph, so it is not graph capture. The NVFP4 swizzled block-scale tensors are tied to their weight-scale siblings in a way this path does not expect. Run with --tp 8 --ep-size 8; note that including the display GPU causes an unrelated OOM, so pin the device list to idle cards.

Why more cards do not (yet) help. vLLM's DeepseekV4ForCausalLM does declare SupportsPP, but the draft model DeepSeekV4MTP does not — and SGLang states the same constraint outright: "Currently DSpark speculative decoding only supports pp_size == 1." With pipeline parallelism unavailable to the drafter and tensor-parallel size restricted to powers of two, DSpark is capped at 8 cards on this architecture no matter how many are installed. Target-only can use pipeline parallelism (we loaded it across 10 cards, 135 GiB resident), though on this no-NVLink host that configuration then hung in shared-memory broadcast — a separate problem, and not one this checkpoint causes.

So: the drafter is present, abliterated, and understood by both stacks — and unexercised here. Acceptance rate and speedup are unmeasured for those two reasons and no other. If you clear either wall, numbers are very welcome in the discussions tab.

Acceptance rate and the resulting speedup on this artifact are NOT measured. Speculative decoding is famously environment-dependent — on this same lab we have measured it losing on CPU MoE inference (batch verification wakes proportionally more experts: 0.658–0.743× on Kimi-K3 GGUF), while GPU reports for DeepSeek V4 class models show clear wins. Measure it on your own box before assuming either. To exercise it you need a stack with DeepSeek MTP support enabled (vLLM ≥ 0.25 or SGLang); our verification above ran the target model only.

Provenance (revisions)

See also ABLITERATION_MANIFEST.json (inherited from the FP8 abliterated source) and the staged SHA256SUMS for every file in this tree.

Quantization recipe (ignore list 1:1)

Producer stamp written into config.json / hf_quant_config.json:

{ "name": "modelopt", "version": "dsv4-nvfp4-experts" }

Hardcoded exclude / HF ignore list (public recipe, not modified):

*.attn.*
*.ffn.shared_experts.*
head
mtp.*

Meaning:

• —Converted: routed MoE experts only → NVFP4 (moe_quant_algo: NVFP4, group size 16).
• —Not converted (passthrough): attention, shared experts, LM head, and all mtp.* (DSpark / multi-token prediction) tensors, remaining in the source mixed formats.

What was actually run (MEASURED bake path)

Command shape (see bake log; device was CUDA for the cast export on this host):

python3 quantize_to_nvfp4.py \
  --amax_path /path/to/amax-nvfp4-experts-abl \
  --source_ckpt /path/to/0731-abliterated-fp8 \
  --output_ckpt /path/to/0731-abliterated-nvfp4 \
  --device cuda \
  --cast_mxfp4_to_nvfp4 \
  --overwrite

Activation PTQ (`ptq.py` / multi-GPU torchrun+NCCL calibration) was NOT run. On the 10×16 GB bake host the full PTQ load path does not fit; input_scale values were stamped from a synthetic constant amax:

• —INPUT_AMAX = 6.0 for every routed expert projection → input_scale = 1/448
• —Weight path under --cast_mxfp4_to_nvfp4 uses the lossless MXFP4→NVFP4 cast (weight amax not used for the casted weights).

Synthetic amax dump: 66 048 keys (33 024 input + 33 024 weight), SHA-256 bcfcd9e53ef3bcf714b0f943448b8f6163c88f9c380297875985293089ecc6b2.

Cast statistics (MEASURED from export log)

Abliteration (inherited; not re-done here)

The NVFP4 cast does not re-apply abliteration. Surgery remains exactly as in the FP8 abliterated source:

• —Method: refusal-direction projection from attention residual writers.
• —Rank 1, λ = 3.5, layers 10–42 inclusive, plus corresponding MTP attention wo_b writers (36 tensors total).
• —Three fixed-point FP8 requantization iterations at abliteration time.
• —Direction SHA-256: 6e4d8a8f3aa9e21795faab2c5b14d29b019acdf2ddbfbd8238430458a5837fe0.

Because attention and MTP are on the ignore / passthrough list, those edited tensors are copied, not requantized, by this bake.

Format and use

Architecture, tokenizer, official message encoding, 1 M context declaration, and attached DSpark tensors come from the 0731 line. Use the included encoding/ directory and the upstream model card for prompt formatting.

Recommended upstream sampling defaults: temperature=1.0, top_p=0.95 for agentic scenarios, top_p=1.0 otherwise.

Serving (NOT verified for this artifact on the bake host)

NVIDIA’s NVFP4 Flash card documents:

• —SGLang, or
• —vLLM ≥ 0.22 (and ≥ 0.25 for DSpark features per project notes).

The bake host’s stock vLLM 0.21.0 is not the verified stack for this checkpoint. Do not treat local 0.21 install as a green light.

Reference NVIDIA deploy notes (for the official NVFP4 sibling, not this abliterated tree): SGLang with TP; vLLM with --kv-cache-dtype fp8 on Blackwell-class hardware.

Validation status

Structural validation (MEASURED on this tree)

Live re-check of the n3 structural gates against the written files:

Pilot (n3a) also showed CPU vs GPU cast byte-identical on a miniature source and full attention/MTP passthrough integrity.

Behavioral validation of this NVFP4 artifact

NOT MEASURED. No refusal gauntlet, no StrongREJECT judge, no capability suite, and no production serve was run against /mnt/stripe/models/0731-abliterated-nvfp4.

Inherited / proxy evidence (about the source abliteration line only)

The FP8 abliterated source card documents an MLX mixed 2/3-bit deployment proxy benchmark (oMLX, 2026-08-01). That evidence:

• —applies to a quantized MLX derivative of the abliterated line,
• —is not a direct measurement of native FP8,
• —is not a measurement of this NVFP4 tree,
• —must not be copied as scores for this release without re-running.

MEASURED vs ESTIMATE (explicit)

MEASURED

• —Source abliterated tree downloaded at HF commit 7d02640c72a2c8127f116d3d1933ddfec5e4c0fa.
• —Bake used public quantize_to_nvfp4.py with --cast_mxfp4_to_nvfp4 and producer dsv4-nvfp4-experts.
• —Ignore list is exactly *.attn.*, *.ffn.shared_experts.*, head, mtp.*.
• —MXFP4→NVFP4 weight cast lossless block rate 100%.
• —Structural gates above all PASS; disk shard total 163.49 GiB.
• —Attention / shared / head / MTP tensors are not NVFP4-rewritten (abliteration residual writers preserved by passthrough).
• —Full per-file SHA-256 inventory of this tree (see SHA256SUMS).

ESTIMATE / synthetic / not measured

• —Activation scales: synthetic INPUT_AMAX=6.0 → input_scale=1/448. Not calibrated on real traffic. Quality impact of synthetic input scales is unknown.
• —Serving stack fit and tokens/s: not measured on this artifact. Plan-level “44 t/s with MTP on 12×16 GB” remains an ESTIMATE / goal, not a result.
• —Refusal / uncensor behavior after NVFP4 cast: not measured.
• —Capability preservation (coding, reasoning, long context): not measured.
• —Parity with `nvidia/DeepSeek-V4-Flash-NVFP4` accuracy tables: not claimed; NVIDIA calibrated activations; this bake did not.

Limitations

• —Abliteration can affect capabilities beyond refusals.
• —Synthetic activation scales may interact poorly with some runtimes or sequences even when weights cast losslessly.
• —DSpark/MTP behavior under NVFP4+passthrough has not been load-tested here.
• —Use access controls appropriate to an abliterated model.

License and attribution

• —Upstream DeepSeek weights: MIT (LICENSE).
• —Refusal direction: drowzeys/keys MIT notice in NOTICE.
• —Quantization recipe: NVIDIA Model-Optimizer (public examples); this card is not an NVIDIA release.
• —Please credit DeepSeek-AI, the abliteration source maintainers, and NVIDIA Model-Optimizer when redistributing or publishing results.

Integrity

Publish only after Ken’s explicit approval. Irreversible Hub upload was not performed by the n4 staging job. Verify against SHA256SUMS after any copy.