CoolFace
Modelpublic

zebulon-prime/Qwen3.8-27B-Dominatrix-MTP-NVFP4

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face
2likes148downloads
Model Card

Qwen3.8-27B-Dominatrix-NVFP4

A mixed-precision NVFP4/FP8 quantization of allura-org/Qwen3.8-27B-Dominatrix, with the base model's BF16 multi-token-prediction (MTP) head grafted back in for speculative decoding.

Not-for-all-audiences. This is a quant of a roleplay/creative-writing finetune with mature content; inherits all behavior and content characteristics of the source model.

Attribution

  • —Finetune: allura-org/Qwen3.8-27B-Dominatrix (Apache 2.0), trained by Allura with Axolotl/Unsloth. All credit for the model's behavior, style, and training data goes to the original authors.
  • —Base model: Qwen/Qwen3.8-27B (qwen3_5, hybrid gated-linear-attention + full-attention, 64 layers, vision-capable).
  • —Quantization recipe format: mirrors RadixArk/Qwen3.8-27B-NVFP4's mixed-precision ModelOpt export (same producer version, same per-layer-type precision assignment), applied here to the Dominatrix finetune instead of the base model.

This repo only contributes the MTP graft and the quantization; no retraining was performed.

What was done

  1. 1.MTP graft. The Dominatrix finetune ships without a multi-token-prediction head (dropped during finetuning). The 15 BF16 MTP tensors (mtp.*) were copied over unmodified from the pre-finetune base checkpoint and merged into the finetune's safetensors shards, since no finetuned MTP head exists. This gives up speculative-decode acceptance rate slightly relative to a co-trained head, but MTP is draft-only — it does not change the served model's output distribution.
  2. 2.Quantization. NVIDIA ModelOpt 0.47.0.dev0, PTQ with method: max calibration on 1024 samples of abisee/cnn_dailymail (seq len 512).
ComponentPrecision
MLP gate_proj / up_proj / down_proj (weights + activations)NVFP4, group size 16
lm_head (weight + input)NVFP4, group size 16
Full-attention self_attn projections (16 of 64 layers)FP8 (weights + activations)
Large linear-attention projections (in_proj_qkv, in_proj_z, out_proj) on the 48 GDN layersFP8 (weights + activations)
KV cacheFP8, constant amax
MTP head, vision tower, embeddings, GDN in_proj_a/in_proj_b/conv1dBF16 (unquantized)

Result: 2194 total tensors — 193 NVFP4 + 208 FP8 quantized layers, 15 MTP tensors carried at full BF16 precision. Structurally verified against the RadixArk reference export: identical per-layer precision map, byte-identical config.json quantization metadata, 0 missing / 0 extra tensors.

Quality: KL divergence vs. the BF16 finetune

Measured directly rather than assumed. Full-vocabulary next-token KL divergence between this NVFP4/FP8 checkpoint and the original BF16 Dominatrix weights, evaluated on 256 held-out cnn_dailymail samples (512 tokens each, ~120k scored token positions), using the exact same calibration recipe/data/seed reconstructed in-process (deterministic max calibration, so the reconstructed quantizer scales match this export's stored scales exactly wherever directly comparable, e.g. the FP8 self_attn projections).

MetricValue
KL(BF16 ‖ NVFP4)0.0575 nats/token (SE 2.9×10⁻⁴, N=120,279 tokens)
Perplexity9.03 (BF16) → 9.49 (NVFP4), +5.1% relative
Top-1 next-token flip rate13.4% (dominated by near-tied/high-entropy positions; not a reliable standalone quality signal on its own — see KL/PPL)

This is a fairly aggressive quantization scheme relative to weight-only alternatives — activations and lm_head are quantized, not just weights, plus an FP8 KV cache — so a ~5% PPL increase is in the expected range for the precision budget, not a red flag.

Serving

Verified serving on SGLang with real FP4/FP8 kernels, FP8 KV cache, and NEXTN speculative decoding against the grafted MTP head:

--speculative-algo NEXTN --speculative-num-steps 3 \
--speculative-eagle-topk 1 --speculative-num-draft-tokens 4