CoolFace
Modelpublic

angrykirc/ThinkingCap-Qwen3.8-27B-INT8-W8A8-imatrix

sourceHugging Faceotherupdated 13h agoView on Hugging Face
1likes
Model Card

ThinkingCap-Qwen3.8-27B — INT8 W8A8 (imatrix)

8-bit INT8 W8A8 quantization of bottlecapai/ThinkingCap-Qwen3.8-27B (55.6 GB BF16 → 30 GB).

Quantization recipe

ComponentSetting
formatcompressed-tensors, int-quantized
weights8-bit int, symmetric, per-channel, static, observer `imatrix_mse`
activations8-bit int, symmetric, per-token, dynamic
targetsLinear
ignoredall model.visual.* blocks · linear_attn.in_proj_a / in_proj_b (+ norm) · lm_head · mtp.*
calibration512 sequences, ~2k tokens each (open-source Python code, half with tool calls), 2048 max seq length, pipeline: sequential
toolingllmcompressor 0.14.0, transformers 5.17.0, torch 2.14.0+cu126

Layout

  • —model-0000X-of-00015.safetensors — quantized weights (400 int8 Linear weights + 400 per-channel weight_scale, everything else BF16)
  • —model_mtp.safetensors — MTP module (15 BF16 tensors) excluded from quantization; load with load_mtp=True in vLLM or ignore if not using speculative decoding
  • —recipe.yaml — exact llmcompressor recipe applied

Usage

transformers

python
from transformers import AutoModelForImageTextToText, AutoProcessor

processor = AutoProcessor.from_pretrained("angrykirc/ThinkingCap-Qwen3.8-27B-INT8-W8A8-imatrix")
model = AutoModelForImageTextToText.from_pretrained(
    "angrykirc/ThinkingCap-Qwen3.8-27B-INT8-W8A8-imatrix",
    dtype=torch.bfloat16, device_map="cuda:0",
)

vLLM

bash
vllm serve angrykirc/ThinkingCap-Qwen3.8-27B-INT8-W8A8-imatrix

Note: for best performance with the linear-attention layers, install flash-linear-attention and causal_conv1d (transformers falls back to reference kernels without them).

Verification

  • —all 400 quantized Linear weights confirmed I8 in checkpoint, 400 per-channel scales
  • —ignored modules confirmed BF16 (vision tower, inproja/b, lm_head, mtp)
  • —smoke test: 104-token greedy generation runs correctly