darioooooo0o/granite-4.2-8b-G-Q5-GGUF

Granite 4.2 8B — G-Q5 mixed quantization + RTX 3060 optimization study


Requests, questions or suggestions? Message me on X: https://x.com/imdariotoo

Two deliverables here:

1. 1.G-Q5 — a custom mixed-precision GGUF of ibm-granite/granite-4.2-8b targeting near-bf16 fidelity at Q5-ish size/speed. Best-in-table accuracy in our testing.
2. 2.A complete quantization/serving study for dense 8B models on consumer 12GB GPUs (RTX 3060): KL-divergence comparisons across all major community quants, KV cache dtype findings, max-context maps, speculative decoding results, and sampling pitfalls. All measured, nothing estimated.

G-Q5 quant recipe

• —Quantized from bf16 (never requantize) with llama.cpp b062ba735.
• —imatrix calibrated on ~600K tokens: wikitext-103 prose + local C++/TS/Python code (~80/20).
• —Rationale: granite's untied embeddings on a 100K vocab are ~20% of params — the biggest accuracy lever. Attention projections carry the arch's unusual 1/128 attention scale.

Measured comparison (RTX 3060 12GB, CUDA, r=3 medians)

Fidelity = KL divergence vs bf16 over 30x512-token chunks of held-out wiki text.

Also tested and rejected:

• —Q4KS-XL (Q4KS + Q5K embd, imatrix): KLD 0.1333, tg 57-58 — strictly dominated by bartowski's IQ4XS.
• —EDGE (IQ2XS middle-layer FFN, IQ3XXS edge FFN, Q8 attention): 4.36G but slower than IQ4_XS (51.6 tg) with catastrophic KLD 0.622. IQ2-class kernels on Ampere MMQ are slow enough to eat the bandwidth savings, and same-top-p collapsed to 71%.

Key finding: official IBM quants are mediocre

The official Q4KM's KLD is nearly 2x worse than bartowski's free imatrix upgrade of the same quant. If you use granite from the official GGUF repo, you're leaving real accuracy on the table.

Serving findings for dense 8B on 12GB

Context limits (full offload, FA on)

KV cost is architecture-determined: 40 full-attention layers x 8 KV heads x 128 headdim = ~80KB/token at f16, ~22KB at q40. Hybrid-attention models (qwen3.5-style: SSM layers between attention) need 1/4 the KV per token — no weight quantization can close that gap.

KV dtype vs speed: no difference at practical depths

At 2.3K-token prompts: q40 KV = q80 KV within noise (pp ~1600, tg ~46 both). KV dtype only matters at deep fill (50K+). Use q80 KV for free quality; drop to q40 only to extend max context.

Agent-workload throughput: expect well below bench numbers

During a real agent session (hermes-core suite), median decode was 29 tok/s vs 50+ on clean llama-bench runs. Causes: growing context per task (attention cost scales with KV depth — long generations ran at 26 vs 32 for short ones) and per-request churn (127 requests in one suite). Plan capacity on the agent number, not the bench number.

Speculative decoding on granite

Same tokenizer != fast draft. Draft speedup requires draft model depth << target depth or a trained single-pass head (DFlash/EAGLE style).

Prefill notes

• —Deep-fill prefill degrades gracefully: ~1635 tok/s at 89K tokens (vs 1750 cold).
• —Decode at depth: 47 tok/s @ 89K fill, 13 tok/s @ 62K fill on 64K server — plan long-context serving accordingly.

Sampling: no significant difference found

We tested greedy (temp=0) vs IBM's official config (temp=1.0, top_p=0.95) across 10 varied prompts: 0/10 spirals for greedy vs 1/10 for official — no significant difference. An earlier single-run observation that temp 0 "causes reasoning loops" did not replicate; the loop we saw was on a heavily-quantized experimental variant, not this one. Use IBM's official config (what the model was tuned with), and give thinking mode a large budget (8K+) for math/code tasks.

Reproduce G-Q5

llama-quantize \
  --imatrix imatrix-granite42-mixed.gguf \
  --tensor-type-file tensor_types_G.txt \
  granite-4.2-8b-bf16.gguf granite-4.2-8b-G-Q5.gguf Q4_K_S 32

<details><summary>tensortypesG.txt</summary>

attn_q=q5_k
attn_k=q5_k
attn_v=q6_k
attn_output=q5_k
ffn_down=q5_k
ffn_gate=q4_k
ffn_up=q4_k
token_embd.weight=q5_k
output.weight=q6_k

</details>

Files

• —granite-4.2-8b-G-Q5-Q5_K_M.gguf — the quant (dominant type Q5_K; mixed per-tensor recipe above)
• —imatrix-granite42-mixed.gguf — importance matrix used (reusable)
• —calibration_mixed.txt — calibration corpus
• —tensor_types_G.txt — per-tensor quantization map

HermesBench agent evaluation (G-Q5)

Ran the hermes-core tool-use suite (13 tasks, sequential, real Hermes agent + tool loop) on G-Q5.

Initial run: 6/13 (46%) — but 5 of the 7 failures were infrastructure, not the model. The llama-server crashed mid-run and 5 tasks burned their retries against a dead port. On rerun with a stable server:

Corrected score: 11/13 (85%). For reference from our July sweep: Qwen3.5-9B UD Q4KXL scored 77% (with MTP spec-decoding, turbo KV, 262K ctx) and Qwen3.6-35B-A3B UD IQ3_XXS scored 85%. Caveats before declaring granite the winner: the Qwen runs used a different serving regime, and this suite is small enough that one task flips the percentage by 8 points. Treat as "granite G-Q5 is in the same tier as the best local models we've tested on this GPU," not a strict ranking.

Tool-calling notes

• —Granite uses a legacy XML tool-call format (<tool_call><function=name><parameter=...>); llama.cpp's --jinja parser handles it correctly — structured tool_calls come back properly.
• —Parallel tool calls require "parallel_tool_calls": true in the request payload (llama.cpp default is off). Without it, only the first call in a turn survives.
• —Single calls, no-tool restraint, and tool-result follow-up turns all work correctly out of the box.

Agent behavior notes

• —Thinking-token heavy: several tasks ran 10-60 min due to verbose reasoning chains, not slowness of the weights.
• —The memory-recall failure is instructive: granite solved the task correctly via a file shortcut but the grader required the memory tool path specifically. Rule-compliance, not capability.

Hardware context

All measurements: AMD Ryzen 9 5950X, RTX 3060 12GB, Fedora 43, llama.cpp b062ba735, CUDA. Localmaxxing runs approved: G-Q5, run ids cmt8zlca200ioli019ukksygh (G-Q5), cmt8zl9b300ikli01o2v42b0q (Q4KS-XL), cmt8wx9zq00ftli014hldgmc6 (official Q4KM baseline).