Modelpublic

IstroSec/ThinkingCap-Qwen3.8-27B-abliterated-NVFP4A16

sourceHugging Faceotherupdated 16h agoView on Hugging Face

0likes

Model Card

ThinkingCap-Qwen3.8-27B-abliterated-NVFP4A16

NVFP4 weight-only quantization (4-bit FP4 weights, bf16 activations) of ThinkingCap-Qwen3.8-27B-abliterated, the uncensored variant of bottlecapai/ThinkingCap-Qwen3.8-27B.

28.6 GB (from 55.6 GB bf16). Plan on a 48 GB-class GPU; 32 GB leaves almost nothing for KV cache. Fastest on Blackwell (native FP4 path); runs on Hopper via the Marlin weight-only kernel (the same support matrix bottlecapai lists for its own NVFP4 weight-only build).

If you have the memory for it, the FP8-DYNAMIC build is closer to lossless. This one is for when 36.8 GB doesn't fit.

What is quantized

Produced with llm-compressor, scheme NVFP4A16: FP4 (E2M1) weights in groups of 16 with FP8 (E4M3) per-group scales and a per-tensor FP32 global scale. Activations stay bf16, so no calibration set is needed and there's no activation-quantization noise.

Quantized: all Linear modules in the 64 decoder layers' MLPs and the 16 full-attention layers' q/k/v/o_proj.

Kept in bf16 on purpose:

Component	Why
`linear_attn.*` (Gated DeltaNet, 48 layers)	4-bit on the recurrent block roughly doubles KL and introduces thinking loops on this architecture
`visual.*`	vision tower and merger
`lm_head`	standard
`mtp.*`	MTP head, re-grafted from bf16 after quantization; its `Linear`s are in `quantization_config.ignore`

The bf16 islands are why this lands at 28.6 GB rather than the ~21 GB of a build that also quantizes DeltaNet (bottlecapai's own NVFP4 weight-only is 21 GB). The DeltaNet block is 5.6B of the 28B parameters. Community measurements on the Qwen3.6-27B sibling showed that 4-bit-quantizing it roughly doubles KL and can produce thinking loops, so those 8 GB are deliberately spent. If size matters more than that risk, bottlecapai's build of the original (censored) model shows the smaller trade-off.

Evaluation

	Refusals (100 harmful)	KL vs. bf16 abliterated
bf16 abliterated (source)	6 / 100	—
FP8-DYNAMIC	{{fp8.refusals}}	{{fp8.kl}}
NVFP4A16 (this repo)	{{nvfp4a16.refusals}}	{{nvfp4a16.kl}}

Method: Heretic --evaluate-model against the bf16 abliterated checkpoint, non-thinking mode. Expect this build to land somewhat above FP8; under ~0.05 KL is the target for a 4-bit build of a reasoning model. {{nvfp4a16.kl_note}}

Provenance chain: original ThinkingCap refuses 97/100 → bf16 abliteration 6/100 at KL 0.065 vs. original → this quantization adds the KL above.

Usage

vLLM (Blackwell for the native FP4 path; Hopper falls back to Marlin automatically)

bash

vllm serve IstroSec/ThinkingCap-Qwen3.8-27B-abliterated-NVFP4A16 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --max-model-len 65536 --gpu-memory-utilization 0.85

Sampling (Qwen3.8 recommendations, which ThinkingCap uses unchanged): thinking mode temperature 1.0, top_p 0.95, top_k 20, min_p 0; non-thinking mode temperature 0.7, top_p 0.8, top_k 20, presence_penalty 1.5. Thinking budget via chat_template_kwargs: {"reasoning_effort": "xhigh"} — xhigh (default, recommended), medium, or low.

Notes:

—Only the 16 full-attention layers have a KV cache; FP8 KV (--kv-cache-dtype fp8_e5m2) is a small win here, not a large one.
—Weight-only FP4 is bandwidth-bound like any weight-only format: single-stream decode is fast, high-batch throughput is where W4A4 (NVFP4) would pull ahead. That variant needs calibration and loses more quality; it isn't published here.

Transformers loads it (dequantized to bf16, so ~56 GB of memory):

python

from transformers import AutoModelForImageTextToText
m = AutoModelForImageTextToText.from_pretrained("IstroSec/ThinkingCap-Qwen3.8-27B-abliterated-NVFP4A16", device_map="cuda")

Not for llama.cpp — compressed-tensors format. GGUF builds are made separately from the bf16 source.

Reproduce

python

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
recipe = QuantizationModifier(
    targets="Linear", scheme="NVFP4A16",
    ignore=["lm_head", "re:.*visual.*", "re:.*linear_attn.*"],
)
oneshot(model=model, recipe=recipe)
# then copy mtp.* from the bf16 checkpoint and add the MTP Linears to quantization_config.ignore

Limitations

Everything from the bf16 card applies: no safety filter, 6/100 residual refusals, thinking mode not separately evaluated. You are the safety layer. Add to that the usual 4-bit caveats: slightly lower accuracy on long multi-step reasoning and code than FP8; if a task is failing here and working on FP8, that's the quantization.

License

PolyForm Small Business License 1.0.0 + BottleCap personal-use grant, inherited from ThinkingCap (see LICENSE). Upstream Qwen materials and the abliteration adapter are Apache-2.0 (see NOTICE). Commercial use beyond the PolyForm terms: contact BottleCap AI.

Credits

bottlecapai (ThinkingCap) · MuXodious (abliteration adapter) · p-e-w/heretic · vllm-project/llm-compressor · Qwen team