michaelw9999/Qwen3.6-27B-NVFP4-MTP-GGUF
Recently re-quantized with the latest version of <A HREF="https://github.com/michaelw9999/advanced-gguf-quantizer">advanced-gguf-quantizer</A> tool.<BR> <BR>MTP included.<BR>
<div> <h3>Qwen3.6-27B RSF NVFP4 GGUF v4</h3>
<p> This latest (6-June-2026) NVFP4 quant of Qwen3.6-27B keeps quality very high, while improving KLD, tail KLD metrics, top-token stability, and probability error compared to the earlier releases. It is faster and smaller in size. It applies my RSF scale fitting technique to the Q_K quants and uses a different tensor mix layout compared to previously. 6-June-2026: changed the MTP tensors to be NVFP4, ~130tk/s tg is now possible in some configurations on 5090. </p> <p>I am experimenting with a smaller verison of this model designed to be used on machines with 16GB of VRAM:<BR> <A HREF="https://huggingface.co/michaelw9999/Qwen3.6-27B-NVFP4-SMALL-MTP-GGUF">Qwen3.6-27B-NVFP4-SMALL-MTP-GGUF</A>.<BR> The SMALL version will be slower than this model but should maintain similar quality, although I am still evaluating it.<BR></BR>
</p> <p>Feedback to improve this is appreciated.</p> Results below on RTX 5090 with Wikitest: <table> <thead> <tr> <th>Model</th> <th align="right">Size GiB</th> <th align="right">pp512</th> <th align="right">tg128</th> <th align="right">pg32768,256</th> </tr> </thead> <tbody> <tr> <td>RSF NVFP4</td> <td align="right">15.27</td> <td align="right">5174.31 ± 1.30</td> <td align="right">76.61 ± 0.18</td> <td align="right">2751.93 ± 23.73</td> </tr> </tbody> </table>
<br>
<table> <thead> <tr> <th>Quality Metric</th> <th align="right">Earlier NVFP4</th> <th align="right">Improved RSF NVFP4</th> <th align="right">Change</th> </tr> </thead> <tbody> <tr> <td>Mean PPL(Q)</td> <td align="right">7.128133 ± 0.047613</td> <td align="right"><b>7.030348 ± 0.046636</b></td> <td align="right"><b>1.37% lower</b></td> </tr> <tr> <td>Mean PPL(base)</td> <td align="right">6.900856 ± 0.045374</td> <td align="right">6.900856 ± 0.045374</td> <td align="right">(Same Base)</td> </tr> <tr> <td>PPL gap vs base</td> <td align="right">0.227277 ± 0.008252</td> <td align="right"><b>0.129492 ± 0.006913</b></td> <td align="right"><b>43.0% smaller gap</b></td> </tr> <tr> <td>Mean PPL(Q) / PPL(base)</td> <td align="right">1.032935 ± 0.001175</td> <td align="right"><b>1.018765 ± 0.000996</b></td> <td align="right"><b>43.0% lower overhead</b></td> </tr> <tr> <td>Mean ln(PPL(Q) / PPL(base))</td> <td align="right">0.032404 ± 0.001137</td> <td align="right"><b>0.018591 ± 0.000978</b></td> <td align="right"><b>42.6% lower</b></td> </tr> <tr> <td>Correlation ln(PPL)</td> <td align="right">98.54%</td> <td align="right"><b>98.91%</b></td> <td align="right">+0.37 pp</td> </tr> </tbody> </table>
<br>
<table> <thead> <tr> <th>KLD Metric</th> <th align="right">Earlier v2</th> <th align="right">Latest RSF NVFP4</th> <th align="right">Change</th> </tr> </thead> <tbody> <tr> <td>Mean KLD</td> <td align="right">0.058781 ± 0.000935</td> <td align="right"><b>0.044590 ± 0.000832</b></td> <td align="right"><b>24.1% lower</b></td> </tr> <tr> <td>99.9% KLD</td> <td align="right">4.869176</td> <td align="right"><b>3.726478</b></td> <td align="right"><b>23.5% lower</b></td> </tr> <tr> <td>99.0% KLD</td> <td align="right">0.591435</td> <td align="right"><b>0.422199</b></td> <td align="right"><b>28.6% lower</b></td> </tr> <tr> <td>95.0% KLD</td> <td align="right">0.165489</td> <td align="right"><b>0.122647</b></td> <td align="right"><b>25.9% lower</b></td> </tr> <tr> <td>90.0% KLD</td> <td align="right">0.096133</td> <td align="right"><b>0.070556</b></td> <td align="right"><b>26.6% lower</b></td> </tr> <tr> <td>Median KLD</td> <td align="right">0.019355</td> <td align="right"><b>0.013642</b></td> <td align="right"><b>29.5% lower</b></td> </tr> <tr> <td>Maximum KLD</td> <td align="right">20.671190</td> <td align="right">24.703529</td> <td align="right">higher</td> </tr> </tbody> </table>
<br>
<table> <thead> <tr> <th>Token Probability Metric</th> <th align="right">Earlier NVFP4</th> <th align="right">Improved RSF NVFP4</th> <th align="right">Change</th> </tr> </thead> <tbody> <tr> <td>RMS Δp</td> <td align="right">6.613 ± 0.060%</td> <td align="right"><b>5.774 ± 0.060%</b></td> <td align="right"><b>12.7% lower</b></td> </tr> <tr> <td>Same-top probability</td> <td align="right">90.450 ± 0.076%</td> <td align="right"><b>91.924 ± 0.071%</b></td> <td align="right"><b>+1.474 pp</b></td> </tr> <tr> <td>Top-flip weight</td> <td align="right">0.009715</td> <td align="right"><b>0.007193</b></td> <td align="right"><b>26.0% lower</b></td> </tr> <tr> <td>Top-prob RMSE</td> <td align="right">0.080946</td> <td align="right"><b>0.069932</b></td> <td align="right"><b>13.6% lower</b></td> </tr> <tr> <td>Entropy RMSE</td> <td align="right">0.248197</td> <td align="right"><b>0.212250</b></td> <td align="right"><b>14.5% lower</b></td> </tr> </tbody> </table>
</div>