xpuenabler/gpt-oss-15.5b-23E-SFT-v6-GpNd-separate-mode-decode-nf4

decode-nf4

NF4 + AWQ quantized variant of xpuenabler/gpt-oss-15.5b-23E-SFT-v6-GpNd-separate-mode-decode. Only the mlp.experts weights were re-quantized; everything else (INT4 attention/dense, FP16 embedtokens / lmhead / router / norms, graph topology) is preserved.

What changed vs xpuenabler/gpt-oss-15.5b-23E-SFT-v6-GpNd-separate-mode-decode

• —48 expert weight chains (24 layers × {gateupproj, downproj}) had their INT8 storage replaced by NF4 with groupsize=64.
• —Per-layer activation-side AWQ multiplies were inserted on the Multiply_1 (Swish) → Multiply_2 link of the MoE block.
• —mlp.router, lm_head, and embed_tokens are excluded from quantization (kept in their original precision).

Quantization parameters

mlp.experts quantization parameters (per-group scales and AWQ activation-side scales) were computed once on the prefill graph using NNCF compress_weights(mode=NF4, group_size=64, awq=True, scale_estimation=True) with 32 calibration samples at static input shape [1, 128]. The resulting NF4 weight chains and AWQ multiplies were then transplanted into the decode graph (which has the same expert weights but different I/O static shapes), so the two graphs share identical quantized parameters.

Memory