malvavisc0/Qwen3.8-9B-gptq-int4

Qwen3.8-9B — GPTQ Int4

[image]

Frontier-scale reasoning. 8.5 GB of VRAM. One consumer GPU.

This is a GPTQ Int4 quantization of `empero-ai/Qwen3.8-9B` — a full-parameter distillation of the Qwen3.8 2.4T A95B teacher into a dense 9B. Every answer opens with a <think> block learned from trillion-parameter-scale reasoning traces. The BF16 original needs ~18 GB; this artifact delivers the same weights in 8.5 GB with a mean per-layer quantization error of 2.3e-5.

Why this model

• —A 2.4T teacher in a 9B student. Math, code, and tool-use CoT distilled from Qwen3.8 2.4T A95B — not self-generated reasoning. MMLU (CoT) jumps +20.5 points over the Qwen3.5-9B base.
• —Int4 without the quality cliff. Calibrated on 256 samples that match the serving distribution — DeepSeek-R1 math traces, competitive programming solutions, and agentic tool-use conversations — rendered through the model's own chat template. Not web text. The calibration sees exactly what production traffic looks like.
• —Full text-stack coverage. All 200 quantizable text modules quantized (24 GatedDeltaNet + 8 full-attention layers, hybrid architecture). The vision tower is passed through untouched.
• —262k native context, preserved.
• —vLLM-ready. Ships with the extended chat template (reasoning effort control, XML tool-call format) and serves with a single vllm serve command.

Where it runs

Rule of thumb: 8.5 GB weights + KV cache. 12 GB of total memory is the practical floor; 16 GB is comfortable; 24 GB+ lets the 262k context breathe.

Quickstart (vLLM — recommended)

pip install "vllm>=0.27"
vllm serve malvavisc0/Qwen3.8-9B-gptq-int4 \
  --quantization gptq_marlin --max-num-seqs 10

--max-num-seqs 10 matters: this is a hybrid GatedDeltaNet/full-attention architecture, and vLLM's Mamba-style cache is happiest with a bounded batch. On 24 GB cards, add --max-model-len 32768 (or less) if you don't need the full 262k.

Then talk to it over the OpenAI API:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="x")
resp = client.chat.completions.create(
    model="malvavisc0/Qwen3.8-9B-gptq-int4",
    messages=[{"role": "user", "content":
        "A snail sits at the bottom of a 10 m well. Each day it climbs "
        "3 m, each night it slips back 2 m. How many days to escape?"}],
    temperature=0.6, top_p=0.95,
    extra_body={"top_k": 20, "max_tokens": 16384},
)
print(resp.choices[0].message.content)

Quickstart (transformers)

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "malvavisc0/Qwen3.8-9B-gptq-int4"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

messages = [{"role": "user", "content": "Prove that √2 is irrational."}]
inputs = tok.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

out = model.generate(
    inputs, max_new_tokens=16384,
    temperature=0.6, top_p=0.95, top_k=20, do_sample=True,
)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

Best practices

• —Sampling: temperature=0.6, top_p=0.95, top_k=20. Greedy decoding causes repetition loops in reasoning models of this class.
• —Token budget: be generous (max_new_tokens ≥ 8192). <think> blocks are long by design — the model deliberates, then answers. Strip the <think>...</think> span for end users.
• —Tool calling: native XML <tool_call> format per Qwen3.5's spec; pass tool definitions via the chat template's tools argument.
• —Reasoning effort: the shipped chat template supports reasoning_effort (low→xhigh) — dial it down for easy questions, up for competition math.

Quantization details

Base revision: 0934f3d2327ff2df2197495278c4c46ae5a56bd9. Produced with `aft` (Aria Finetuner).

Sibling artifacts

• —`empero-ai/Qwen3.8-9B` — BF16 original + benchmark table
• —empero-ai's GGUF builds — for llama.cpp / CPU / Metal

License

Apache-2.0, inherited from the Qwen3.5-9B base via the upstream distillation. Credit to Empero for the model and the Qwen team for the architecture.