ProCreations/grug-9b-qat-q4

grug-9b-qat-q4

grug put 9b bird in four-bit cave during training. bird feel rounding rock before final squish. then grug make real Q4KM GGUF. this not normal squish. this QAT recovery.

QAT start from full grug-9b, never train old GGUF, never requantize quant rock. grug fake-quant text linear weight as asymmetric int4, group 32, straight-through gradient. train 3,628,102 token from grug-think. full QAT overcorrected, so grug keep 25% QAT move and 75% original anchor. fixed eval gate choose before release. then export fresh BF16 GGUF and quantize one time to Q4KM.

numbers. same cave, same harness

both rocks run same llama.cpp build, greedy, same prompt. control = ordinary grug-9b-Q4_K_M. QAT = this rock.

grug reasoning format stay same: short grug brain inside <think>...</think>, normal say-word outside, XML tool call untouched. grug measure presence and length above, not merely hope.

grug honest: QAT help model adapt to 4-bit rounding. QAT not magic promise. numbers above say what happen on these probe. hardest real repo task may act different.

rock

grug-9b-qat-Q4_K_M.gguf — about 5.6 GB. good default for 8 GB VRAM cave.

llama-cli -hf ProCreations/grug-9b-qat-q4:Q4_K_M

need recent llama.cpp with qwen3_5 support. thinking live in <think> tag and stay short on purpose. tool call use model XML format.

recipe. grug show work

• —source: ProCreations/grug-9b
• —data: ProCreations/grug-think
• —QAT: full text-stack linear weights, fake int4 asymmetric group size 32
• —training: 684 optimizer step, 3,628,102 token, LR 2e-06
• —release blend: 25% QAT checkpoint + 75% original checkpoint. grug show this because honest rock better than secret rock
• —target export: llama.cpp Q4KM, quantized from fresh BF16 conversion
• —vision tower not in GGUF. text bird only

training config and measured JSON live in this repo. grug no hide recipe behind bush.