Modelpublic

wasifb/ThinkingCap-Qwen3.8-27B-AutoRound-W4A16

sourceHugging Faceotherupdated 1d agoView on Hugging Face

3likes869downloads

Model Card

ThinkingCap-Qwen3.8-27B — AutoRound INT4 (W4A16)

An INT4 AutoRound quantization of bottlecapai/ThinkingCap-Qwen3.8-27B, BottleCap AI's reasoning fine-tune of Qwen/Qwen3.8-27B. It is 19.0 GB and keeps the MTP head for self-speculative decoding.

BottleCap publishes FP8, NVFP4 and GGUF builds, but no INT4 AutoRound. This one exists so the model can run on two 24 GB cards with the same tiers as the base Qwen3.8-27B in club-3090.

Quantization


Scheme	W4A16: int4, symmetric, group size 128, `auto_round:auto_gptq` packing
Kept in BF16	`lm_head`, embeddings, vision tower, GDN `linear_attn.in_proj_a` / `in_proj_b`, `mtp.fc`, all norms
MTP head	`mtp.fc` in BF16; the MTP layer's 7 linear weights in int4 (round-to-nearest, not tuned)
Tool	auto-round 0.14.2
Tuning	200 iterations, 512 samples, seqlen 2048, `enable_alg_ext=True`
Calibration	auto-round's default (`NeelNanda/pile-10k`)

The recipe is also stored in club3090_build_recipe.json.

The packed layout is tensor-for-tensor identical to Frozenlock/Qwen3.8-27B-int4-AutoRound: the same quantized and BF16 module families, and the same MTP tensors. So it should load anywhere that checkpoint does. Only the tuning differs: 512 samples with alg_ext here, where that card says "default tuning".

Serving (vLLM)

bash

vllm serve wasifb/ThinkingCap-Qwen3.8-27B-AutoRound-W4A16 \
  --quantization auto_round --dtype bfloat16 --tensor-parallel-size 2 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":4}' \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder

Sampling follows the base model: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0. Thinking effort is set with chat_template_kwargs: {"reasoning_effort": "xhigh"}, which is the template's default.

Known upstream issues on stock vLLM, each fixed in club-3090's composes:

—MTP with prefix caching on Qwen3-Next hybrids can corrupt the recurrent state (vllm#43559; fix vllm#48375 is still open). club-3090 vendors the fix. On stock vLLM, disable prefix caching when MTP is on.
—W4A8 (VLLM_MARLIN_INPUT_DTYPE=int8, roughly +40–50% prefill in our Ampere measurements) doesn't work on stock vLLM for AutoRound checkpoints. The setting is silently ignored (vllm#48904), and negative group scales corrupt the output (vllm#48905). Both are patched in club-3090. On stock vLLM, serve it as W4A16.

Validation

Tested on 2× RTX 3090 (PCIe, 230 W cap) with vLLM v0.30.0 through club-3090's dual-fast configuration: TP=2, W4A8, MTP n=4, fp8 KV, 262K context. Compared against Frozenlock's INT4 of the base Qwen3.8-27B on the same rig and image, the same day.

	this quant
verify-full	pass
Decode, narrative / code (tok/s)	74.0 / 101.0
Prefill @10K / @90K (tok/s)	1,720 / 1,246
MTP acceptance length, n=4	3.6–5.0 (bench); 2.7–4.0 throughout the 20K run
20,000-token forced generation	no drafter collapse, 0 restarts

These are decode and serving checks. The quality evaluation, club-3090's 8-pack, is pending, and this card will be updated with it.

License

ThinkingCap is licensed under the PolyForm Small Business License 1.0.0, plus BottleCap AI's personal-use permission; see LICENSE. This quantization is a derivative and is distributed under the same terms. Commercial use by organizations that don't qualify as a small business needs a commercial license from BottleCap AI.

ThinkingCap is derived from Qwen materials, which are licensed under Apache-2.0. See NOTICE and LICENSE-Apache-2.0-Qwen.txt.

All credit for the model goes to BottleCap AI and the Qwen team. This repository only adds the quantization.