jan1k/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-NVFP4-GGUF

Qwen3.8-27B-Uncensored-HauhauCS-Aggressive — NVFP4 imatrix GGUF

NVFP4 quantisations of HauhauCS/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP-GGUF, built with michaelw9999/advanced-gguf-quantizer (a llama.cpp fork focused on NVFP4/MXFP6 quantization).

Three calibration variants, each with MTP and noMTP builds:

All six share the same source, protection policy, and architecture profile (qwen35dense). They differ only in the calibration data used to compute NVFP4 input scales (or, for v2, the absence of it).

Source

Requantised from Q8KP, not BF16. Tensors that were q8 in the source go through one extra rounding step; bf16 source tensors requantise cleanly.

Imatrix variants

NVFP4 uses per-block input scales. Without an imatrix those scales fall back to identity (all 1.0). An imatrix pass measures how much each weight influences activations on real text, and the quantiser spends the input scale budget where it matters.

v5 — plain text. Bartowski's v5 calibration set, ~1.7 MB of plain prose. Broad language coverage, no chat-template structure.

v6 — chat-template rendered. Bartowski prose plus 173 conversations (137 base + 36 tool-calling) rendered through the target model's chat template using froggeric/Qwen-Fixed-Chat-Templates v22.5. Rendering preserves the special/control tokens and role structure the model sees at inference time, so the activation statistics better match chat/tool-use workloads. llama-imatrix was run with --parse-special so the rendered special-token markup is treated as real tokens, not literal text.

For chat, reasoning, or tool calling: use v6. For plain completion or broad text work: v5 is fine. v2 (no imatrix) is the baseline.

Tensor protection policy

The protection policy follows Luffy's per-tensor quant policy, mapped from the MoE Qwen 3.6 35B (ffn_down_exps) to this dense model (ffn_down).

Implemented with anchored --tensor-type regex overrides (last-match-wins, so patterns are anchored with ^...$ to avoid partial matches like ssm_a matching ssm_alpha).

F16 singular-collapse protection:

Note: blk.0.attn_gate / attn_qkv were q8_0 in the source, so F16 here stores dequantised q8 values — keeps q8-level fidelity instead of dropping to 4 bits, but is not true F16 precision.

F32 architecture-specific protection (not in the original Luffy policy; derived from runtime failures during testing of the sibling Genesis V1 build):

Without the 1D-norm F32 protection, the quantiser emits 198 extra .scale / .input_scale tensors that the FastMTP loader at the investigated revision does not declare, causing a tensor-count mismatch (expected 1866, got 2064).

Forced NVFP4 (do not push lower, collapses):

Everything else eligible takes NVFP4.

Tensor mix

• —general.file_type = 39 (NVFP4)
• —qwen35.block_count = 65 (64 + 1 MTP)
• —qwen35.nextn_predict_layers = 1
• —No unexpected 1D norm .scale tensors

noMTP derivatives strip the 31 blk.64.* tensors and the qwen35.nextn_predict_layers metadata key, decrement block_count 65 → 64, and end at 1835 tensors / 14.46 GB.

FastMTP

The MTP variants are designed for HauhauCS FastMTP using the prebuilt draft sidecar from the same repo. The sibling Genesis V1 build was tested with the same architecture and FastMTP loader; this build shares the identical tensor layout and protection policy.

Usage

llama-cli -m Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-NVFP4-v2-imx-v6-froggeric.gguf \
  -ngl 99 -c 4096 --temp 0.6 --top-k 20 --top-p 1.0 --min-p 0.0

Sampling follows Genesis guidance (temp 0.6, topk 20, topp and min_p disabled). For code, also set --repeat-penalty 1.0 — code repeats identifiers by nature, and any penalty pushes the model off the identifier it just chose.

Give it room to think

This model reasons at length before answering. It can spend 900, even 2600 tokens inside the thinking block without reaching its final answer. Budget generously (-n 4096 or more) for anything non-trivial.

FastMTP

llama-server \
  -m Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-NVFP4-v2-imx-v6-froggeric.gguf \
  --spec-draft-model Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-FastMTP-32K.gguf \
  --spec-draft-ngl all \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --spec-draft-p-min 0 \
  --ctx-size 32768 \
  --parallel 1 \
  --batch-size 2048 \
  --ubatch-size 512 \
  --n-gpu-layers all \
  --split-mode none \
  --flash-attn on \
  --jinja

Use the MTP file for FastMTP, not the noMTP derivative. The noMTP file is for ordinary inference where the MTP/NextN head is not wanted.

Hardware

• —Blackwell (RTX 50xx): native FP4 path, fastest. Build llama.cpp with BLACKWELL_NATIVE_FP4 = 1.
• —Ampere (RTX 30xx): NVFP4 inference works via fallback kernels. The F32 SSM protections prevent a CUDA assert in the patched SSM convolution kernel on this architecture.
• —Quantisation was done CPU-only because the Ampere CUDA NVFP4 encoder is inefficient (hangs/spins at 99% GPU usage with ~300MB VRAM).

Reproducibility

v5 imatrix

llama-imatrix \
  -m Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-Q8_K_P.gguf \
  -f calibration_datav5.txt \
  -o imatrix_v5.dat \
  --output-format dat \
  -ngl all -ts 3,1 -b 2048 -ub 512 -t 6

v6 imatrix

Conversations rendered through froggeric v22.5 with transformers.AutoTokenizer.apply_chat_template(..., tools=tools, add_generation_prompt=False, tokenize=False), concatenated with prose, fed to llama-imatrix with --parse-special:

llama-imatrix \
  -m Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-Q8_K_P.gguf \
  -f calibration_v6_froggeric_rendered.txt \
  -o imatrix_v6_froggeric.dat \
  --output-format dat \
  -ngl all -ts 3,1 -b 2048 -ub 512 -t 6 \
  --parse-special

Quantisation

All variants use the same protection policy and CPU-only quantise command, differing only in the imatrix file (or its absence for v2):

llama-quantize \
  --allow-requantize --mode fast \
  --tensor-type .*=nvfp4 \
  --tensor-type '^blk\..*\.attn_norm\.weight$=f32' \
  --tensor-type '^blk\..*\.post_attention_norm\.weight$=f32' \
  --tensor-type '^blk\..*\.attn_q_norm\.weight$=f32' \
  --tensor-type '^blk\..*\.attn_k_norm\.weight$=f32' \
  --tensor-type '^blk\..*\.ssm_norm\.weight$=f32' \
  --tensor-type '^blk\..*\.nextn\..*norm\.weight$=f32' \
  --tensor-type '^output_norm\.weight$=f32' \
  --tensor-type '^blk\..*\.ssm_conv1d\.weight$=f32' \
  --tensor-type '^blk\..*\.ssm_dt\.bias$=f32' \
  --tensor-type '^blk\..*\.ssm_a$=f32' \
  --tensor-type '^blk.0.ssm_out.weight$=nvfp4' \
  --tensor-type '^blk.1.attn_gate.weight$=nvfp4' \
  --tensor-type '^blk.1.attn_qkv.weight$=nvfp4' \
  --tensor-type '^blk.0.attn_gate.weight$=f16' \
  --tensor-type '^blk.0.attn_qkv.weight$=f16' \
  --tensor-type '^blk.0.ffn_down.weight$=f16' \
  --tensor-type '^blk.13.ffn_down.weight$=f16' \
  --imatrix imatrix_v6_froggeric.dat \
  Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-Q8_K_P.gguf \
  Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-NVFP4-v2-imx-v6-froggeric.gguf \
  Q8_0 6

general.file_type is patched to 39 after quantisation.

noMTP derivatives

strip_mtp.py removes the 31 blk.64.* tensors and the qwen35.nextn_predict_layers metadata key, and decrements qwen35.block_count 65 → 64.

Credits

• —Base model and finetune: HauhauCS
• —Per-tensor protection policy: LuffyTheFox
• —Chat template for v6 calibration: froggeric/Qwen-Fixed-Chat-Templates v22.5
• —Quantiser: michaelw9999/advanced-gguf-quantizer
• —Calibration corpus: Bartowski v5 plain text; v6 prose + conversations rendered through the target chat template