Modelpublic

Hob-forge/Qwen3.5-4B-Instruct-GGUF

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

1likes967downloads

Model Card

Qwen3.5-4B-Instruct — Hob Forge Edition (GGUF)

Qwen's excellent 4B, set up correctly for small GPUs and modest RAM — with everything measured, nothing projected. This is a Hob Forge edition: we didn't train this model (all credit to Qwen, Apache-2.0); we quantized it with a current toolchain, verified the chat template and tool-calling actually work, measured speed / memory / quality-loss for every file, and wrote the run guide we wished existed. If it's in a table below, we ran it on real hardware.

The headline you won't find on other GGUF pages: this is a hybrid-attention architecture (8 full-attention layers + linear-attention DeltaNet + a 1-layer MTP block) — its KV cache is ~5× smaller than a dense 4B. Measured: 8K context costs 256 MiB; 32K costs 1 GiB. Long context on tiny hardware is this model's superpower.

Which file? (measured on RTX 5070, `-ngl 99`, llama.cpp b368b24c)

File	Size	Gen speed tg128	Perplexity (wikitext-2, 120 chunks)	Note
Q4_K_M ⭐	2.58 GiB	138 t/s	10.030 ±0.157	recommended default (imatrix)
Q5KM	2.93 GiB	131 t/s	10.009 ±0.158	quality step up
Q6_K	3.31 GiB	120 t/s	9.887 ±0.155	near-lossless
Q8_0	4.28 GiB	102 t/s	9.832 ±0.154	reference quality
IQ4_XS	2.39 GiB	45 t/s ⚠	10.094 ±0.159	smallest, but i-quant dequant is ~3× slower on RTX cards — only pick this if the last 200MB matters more than speed
F16	8.42 GiB	—	9.871 ±0.155	conversion source

F16 baseline shows the whole ladder loses almost nothing: Q4KM sits +0.16 PPL from F16, inside ~1σ. imatrix (for Q4KM and IQ4_XS) computed from a 2MB wikitext-2-train slice, 250 chunks — the calibration file ships in this repo (calib/).

Memory budget (measured, not projected)

Architecture-fixed overheads (identical for every quant): KV cache + 50 MiB recurrent-state

—~70–96 MiB compute buffer.

Context	KV cache	Total @ Q4_K_M	Total @ Q8_0	Fits 4GB?	Fits 8GB?
4K	128 MiB	~2.8 GiB	~4.5 GiB	✅ Q4	✅
8K	256 MiB	~2.9 GiB	~4.7 GiB	✅ Q4	✅
16K	512 MiB	~3.2 GiB	~4.9 GiB	✅ Q4	✅
32K	1 GiB	~3.7 GiB	~5.4 GiB	⚠ tight	✅

Yes — 32K context, whole model, under 4GB at Q4KM. Every number from llama.cpp's own allocator logs on our hardware.

Run it

bash

# ollama
ollama run hf.co/Hob-forge/Qwen3.5-4B-Instruct-GGUF:Q4_K_M

# llama.cpp (recent build required — this arch needs 2026 support)
llama-cli -m Qwen3.5-4B-Instruct-Q4_K_M.gguf -st -p "your prompt" -ngl 99 -c 8192

Sampling (Qwen's recommendations, verified here): thinking mode temp 0.6, top_p 0.95, top_k 20; non-thinking temp 0.7, top_p 0.8, top_k 20. This model thinks by default — it emits reasoning before answering. Turn it off: think:false (top-level, ollama API) / enable_thinking=False (transformers) / strip <think> blocks client-side for llama.cpp.

Full walkthrough — zero-to-first-tool-call on an 8GB-class GPU and a standard desktop, with the troubleshooting we earned building this — in `RUNNING.md`.

Provenance & method

—Base: Qwen/Qwen3.5-4B (Apache-2.0) — untouched weights, full multimodal snapshot converted text-only.
—Toolchain: llama.cpp convert (build 2026-08-15) + quantize (build 2026-08-12). Chat template verified by rendering tests (system/no-system/tools × both); tool-calling exercised with a real call.
—Evals here measure our files (quantization quality), not the model's intelligence — for capability benchmarks see Qwen's card. PPL runs used identical chunks across all quants.
—No training data involved; nothing to decontaminate. No abliteration anywhere in lineage.

Limits

A 4B is a 4B: strong for its size at chat, coding assistance, and tool use; not a frontier model. IQ4_XS speed caveat above. Vision components of the base are not included (text-only GGUFs). MTP block included in F16 but speculative decoding needs runtime support.