jan1k/Swift-Qwen3.8-27B-Genesis-NVFP4-GGUF

Swift-Qwen3.8-27B-Genesis — NVFP4 GGUF

NVFP4 quantisation of LuffyTheFox/Swift-Qwen3.8-27B-Genesis-F16-GGUF, built with advanced-gguf-quantizer (a llama.cpp fork focused on NVFP4 quantization).

Base model: ukisai/Swift-Qwen3.8-27B-GGUF with Genesis tensor repair by LuffyTheFox.

Files

v4 — Recommended (inline scales, LM Studio compatible)

v4 uses --nvfp4-inline-scales-only to emit native inline UE4M3 scales without separate .scale/.input_scale tensors. Required for LM Studio / Pelican and other runtimes that do not support the extended NVFP4 scale tensor contract.

Imatrix variants will follow after the plain v4 is confirmed working in target runtimes.

Source

Single-step F16 → NVFP4 — the F16 source gives the NVFP4 encoder clean data with no intermediate quantization noise. llama-quantize reads the split source directly via ...-00001-of-00003.gguf.

Tensor mix (v4)

Tensor protection policy

F16 singular-collapse protection:

F32 architecture-specific protection:

• —blk.*.attn_norm.weight, blk.*.post_attention_norm.weight
• —blk.*.attn_q_norm.weight, blk.*.attn_k_norm.weight
• —blk.*.ssm_norm.weight, blk.*.nextn.*.norm.weight
• —output_norm.weight
• —blk.*.ssm_conv1d.weight, blk.*.ssm_dt.bias, blk.*.ssm_a

Forced NVFP4 (do not push lower):

Usage

llama-cli -m Swift-Qwen3.8-27B-Genesis-NVFP4-v4.gguf \
  --mmproj mmproj-Swift-Qwen3.8-27B-F16.gguf \
  --jinja -c 131072 -ngl 99

For the MTP variant, use Swift-Qwen3.8-27B-Genesis-NVFP4-v4-noMTP.gguf when the runtime lacks FastMTP support, or the MTP file for speculative decoding.

• —Set K cache and V cache quantization to F16
• —Vision support requires the mmproj-Swift-Qwen3.8-27B-F16.gguf from the source repository

Hardware

• —Blackwell (RTX 50xx): native FP4 path, fastest
• —Ampere (RTX 30xx): NVFP4 inference works via fallback kernels
• —Quantisation was done CPU-only (Ampere CUDA NVFP4 encoder is unreliable)

Reproducibility

# F16 split source -> NVFP4 v4 (inline scales only, CPU-only)
CUDA_VISIBLE_DEVICES="" llama-quantize \
  --allow-requantize --mode fast --nvfp4-inline-scales-only \
  --tensor-type '.*=nvfp4' \
  --tensor-type '^blk\..*\.attn_norm\.weight$=f32' \
  --tensor-type '^blk\..*\.post_attention_norm\.weight$=f32' \
  --tensor-type '^blk\..*\.attn_q_norm\.weight$=f32' \
  --tensor-type '^blk\..*\.attn_k_norm\.weight$=f32' \
  --tensor-type '^blk\..*\.ssm_norm\.weight$=f32' \
  --tensor-type '^blk\..*\.nextn\..*norm\.weight$=f32' \
  --tensor-type '^output_norm\.weight$=f32' \
  --tensor-type '^blk\..*\.ssm_conv1d\.weight$=f32' \
  --tensor-type '^blk\..*\.ssm_dt\.bias$=f32' \
  --tensor-type '^blk\..*\.ssm_a$=f32' \
  --tensor-type '^blk.0.ssm_out.weight$=nvfp4' \
  --tensor-type '^blk.1.attn_gate.weight$=nvfp4' \
  --tensor-type '^blk.1.attn_qkv.weight$=nvfp4' \
  --tensor-type '^blk.0.attn_gate.weight$=f16' \
  --tensor-type '^blk.0.attn_qkv.weight$=f16' \
  --tensor-type '^blk.0.ffn_down.weight$=f16' \
  --tensor-type '^blk.13.ffn_down.weight$=f16' \
  Swift-Qwen3.8-27B-F16-00001-of-00003.gguf \
  Swift-Qwen3.8-27B-Genesis-NVFP4-v4.gguf NVFP4 6

Credits

• —Base model: ukisai/Swift-Qwen3.8-27B
• —Genesis tensor repair + F16 GGUF: LuffyTheFox
• —NVFP4 quantisation: jan1k
• —Quantiser: advanced-gguf-quantizer