stevelikesrhino/Qwen3.6-27B-Q8-NVFP4-GGUF

Credits

• —Used mmangkad/Qwen3.6-27B-NVFP4 as base model (Thank you for doing ModelOpt quant!)

Notes

• —Another mixed precision quant: ssm_out is bf16, other ssm norms and weights are in F32.
• —Wv and attention output is in bf16.
• —FFN tensors are in nvfp4, with modelopt's input scale / weight scale kept. Currently llama.cpp doesn't have the best implementation for correctly derived nvfp4, but it's only a small loss in perplexity.
• —Resulting BPW is 6.98, for 5090 you can do 130k KV cache at 16 bit.
• —Recommend FP16 KV cache. Do NOT follow previous recommendation for BF16 KV cache!!!

Flow

• —converthfto_gguf with mmangkad's nvfp4 checkpoint
• —generate "keep" list to preserve tensors
• —llama-quantize to q8_0 for the rest of the tensors

NVFP4

NVFP4 related tensors are untouched: weight scales for proper fp4 scaling are kept unchanged.

Image

For image, get any of the mainstream mmproj-bf16 gguf.