CoolFace
Modelpublic

rockerBOO/qwen3-vl-8b-thinking-heretic-nvfp4-convrot-int8

sourceHugging Faceapache-2.0updated 28d agoView on Hugging Face
0likes
Model Card

Qwen3-VL-8B-Thinking-heretic-1refusal — NVFP4 + ConvRot INT8

Quantized from Kizzington/Qwen3-VL-8B-Thinking-heretic-1refusal, itself a Heretic-decensored derivative of Qwen/Qwen3-VL-8B-Thinking.

Files

FileSizeWhat it isDerived from
qwen3-vl-8b-thinking-heretic_nvfp4_convrot_int8.safetensors8.2 GBText decoder: NVFP4 (bulk MLP) + ConvRot INT8 (attention). Vision tower, embeddings, lm_head, and the first/last decoder blocks kept at source BF16.Consolidated BF16 source checkpoint (~18 GB)

Hardware requirements

Requires a Blackwell-class GPU (SM ≥ 10.0) for NVFP4 inference. The INT8 ConvRot layers have broader compatibility on their own, but the file as a whole needs Blackwell because of the NVFP4 portion.

Verified loading and generation (both plain and thinking mode) via ComfyUI's CLIPLoader (type: krea2) + TextGenerate nodes on an RTX 5070 Ti.

Quantization method

Produced with `convert_to_quant` (ctq), NVFP4 + ConvRot INT8 tier, per a role-based layer split:

  • —NVFP4 (bulk, default): model.language_model.layers.N.mlp.{gate,up,down}_proj.weight for the middle 34 decoder blocks (all except the first and last).
  • —INT8 ConvRot (secondary): model.language_model.layers.N.self_attn.{q,k,v,o}_proj.weight for the same middle 34 decoder blocks.
  • —Kept at source BF16 (excluded from quantization): the entire vision tower (model.visual.* — patch embed, ViT blocks, DeepStack merger), model.language_model.embed_tokens.weight, lm_head.weight, model.language_model.norm.weight, and every tensor in decoder blocks 0 and 35 (first/last, kept high-precision for structural-position sensitivity).

Exact command:

bash
ctq -i model_consolidated.safetensors \
    -o qwen3-vl-8b-thinking-heretic_nvfp4_convrot_int8.safetensors \
    --nvfp4 --fallback fp8 \
    --layer-config qwen3vl_8b_layer_config.json \
    --comfy_quant --save-quant-metadata

The layer-config was generated from the checkpoint's own tensor header by `build_qwen3vl_layer_config.py`.

Verified per-layer format distribution in the output's own _quantization_metadata: 102 nvfp4 layers, 136 int8_tensorwise (ConvRot) layers — matching the intended split exactly.

Quality has not been separately assessed beyond spot-checking generation coherence (a factual question and a math "show your reasoning" prompt in thinking mode both produced correct, coherent output) — no formal benchmark/perplexity comparison against the BF16 source has been run.

License

Apache License 2.0, inherited from the source model (Qwen/Qwen3-VL-8B-Thinking and the Kizzington/Qwen3-VL-8B-Thinking-heretic-1refusal derivative). See the Apache 2.0 license text for full terms.