Modelpublic

koloved/Qwen3.8-27B-Heretic-NVFP4-NInfer

sourceHugging Faceupdated 20d agoView on Hugging Face

1likes508downloads

Model Card

Qwen3.5-27B Heretic NVFP4 (NInfer artifact)

Mixed-precision NVFP4 .ninfer artifact of the Heretic (Uncensored) variant of Qwen3.5-27B, optimized for single-GPU inference on NVIDIA RTX 5090.

What this is

This is a quantized inference artifact (.ninfer format) — not safetensors, not GGUF. It is produced by the NInfer converter and runs exclusively with the NInfer inference engine.

Source model

Based on llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved — a BF16 Heretic fine-tune of Qwen3.5-27B with native MTP (Multi-Token Prediction) support preserved.

Quantization recipe

Mixed-precision quantization applied via custom converter:

Component	Precision	Details
MLP layers 0–55	NVFP4	Block-quantized 4-bit with E4M3 block scales, global divisor = 2688/amax
Attention layers	FP8	Row-wise E4M3 scales
GDN layers	FP8	Row-wise E4M3 scales
LM head	FP8	Row-wise E4M3 scales
MLP layers 56–63	FP8	Row-wise E4M3 scales
Embeddings	BF16	Unquantized
MTP head	BF16	Unquantized
Draft head	BF16	Unquantized

Artifact structure

ninfer-serve qwen3_8_27b_nvfp4-heretic.ninfer \
  --max-context 262144 \
  --kv-capacity auto \
  --kv-dtype int8 \
  --max-concurrency 3 \
  --spec mtp --draft-tokens 3 --lm-head-draft \
  --vision \
  --preserve-thinking

Performance

—Artifact size: 18.0 GB
—VRAM usage: ~17 GB (weights) + ~13 GB KV cache headroom
—Decode speed: ~150–190 tok/s (MTP speculative decoding, ~80% acceptance)
—Prefill speed: ~2000–4000 tok/s
—Max context: 262,144 tokens

How this was made

1.Downloaded BF16 Heretic source (52 GB, 13 shards) from HuggingFace
2.Quantized with custom Python quantizer (RTN — Round-To-Nearest):
3.FP8: amax / 448 divisor convention
4.NVFP4: 2688 / amax global divisor, block-scale per 128×4 tile, swizzled encoding
5.Gate/Up projection pairs share global scale from concatenated amax
6.Converted to .ninfer format using NInfer converter
7.Verified against reference artifact (Ostfralla/Qwen3.8-27B-NVFP4-NInfer) — 0 alignment mismatches

Requirements

—GPU: NVIDIA RTX 5090 (Blackwell sm_120a)
—CUDA: 13.1+
—Engine: NInfer (build from source)
—OS: Linux (64-bit)

Quick start

bash

# Build NInfer
git clone https://github.com/Neroued/ninfer.git
cd ninfer
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

# Download this artifact
hf download koloved/Qwen3.8-27B-Heretic-NVFP4-NInfer \
  qwen3_8_27b_nvfp4-heretic.ninfer \
  --local-dir models

# Run server
./build/apps/ninfer-serve models/qwen3_8_27b_nvfp4-heretic.ninfer \
  --max-context 262144 \
  --kv-capacity auto \
  --kv-dtype int8 \
  --max-concurrency 3 \
  --spec mtp --draft-tokens 3 --lm-head-draft \
  --vision \
  --preserve-thinking

Credits

—Base model: Qwen/Qwen3.5-27B (Apache-2.0)
—Heretic fine-tune: llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved
—Inference engine: Neroued/ninfer (Apache-2.0)
—FP8/NVFP4 reference: unsloth/Qwen3.8-27B-NVFP4

License

Apache-2.0 (inherited from base model and NInfer engine).