kingjones777/Gemma-4-12B-it-ROCmFP4-GGUF

Gemma-4-12B-it — ROCmFP4 / ROCmFPX GGUF

AMD-native FP4 / FP8 GGUF builds of google/gemma-4-12b-it, for the ROCmFPX fork of llama.cpp on RDNA3.5 / Strix Halo (gfx1151). These tensor types do not exist in mainline llama.cpp — you need a ROCmFPX-capable build to load them.

Multimodal (mmproj included) and shipped with the MTP draft head, which is worth 1.46× here (see below).

Variants — all five in this repo

Decode measured on an idle Ryzen AI MAX+ 395 (Strix Halo, 128 GB, ROCm 7.2.4), -ngl 999, -c 4096 -fa on -fit off -np 1, 300-token generations, no draft head — see the MTP section for the accelerated figures.

⚠️ Tied embeddings — --output-tensor-type is a silent no-op here

gemma-4-12b-it sets tie_word_embeddings: true, so there is no `output.weight` tensor. Passing --output-tensor-type does nothing at all; `--token-embedding-type` is the only head protection that applies. With a 262144-token vocabulary at n_embd 3840 the embedding is a large share of the file, which is why the 4-bit lands at 4.68 BPW rather than ~4.0.

Measured — not estimated

\ = 5 samples rather than 12. Those two rows each had a single low sample, which a median absorbs; the 4-bit rows were re-run at 12 samples because two* low samples of five moved the number and inverted the FAST-vs-COHERENT ordering.

Correctness is 17×23 → 391 · capital of Japan → Tokyo · days in 2024 → 366, asserted against content + reasoning and recorded with finish_reason. This model is a reasoner: it routinely returns an empty content with the answer in reasoning_content, so a harness that only reads content will score correct answers as failures.

Size integrity

Every artifact's on-disk size exceeds its dry-run projection by a constant ~%s MiB header delta (spread across all five: 0.0075 MiB). A varying delta is the signature of a truncated write; a constant one is just the header.

### ⚠️ If the MTP draft command crashes on your build The separate-model draft-mtp path has known bugs in the legacy charlie12345/ROCmFPX line — reported on Windows 11 / gfx1151 / HIP SDK 7.2 (five stacked bugs, ending in an h-row width mismatch in the draft's embedding buffer). Fix: build the official repo instead — no patches needed. ``bash git clone https://github.com/ROCmFPX/ROCmFPX.git ` The MTP path was reworked there (unified nembdout row widths, thnextn reset in llmgraphresult::reset(), ctx_other` wired centrally), which covers the whole chain. Reported and verified on that configuration: 24–31 tok/s, coherent output, tool calling working. Legacy-line patch: PR #109. Linux builds on the legacy line are not known to be affected.

⭐ Speculative decoding (MTP) — big speedup, with one real caveat

mtp-gemma-4-12b-it-Q8_0.gguf ships in this repo. Unlike the E2B/E4B members of this family, the draft head is a substantial win here — read the caveat below before enabling it.

llama-server -m gemma-4-12b-it-Q4_0_ROCMFP4_COHERENT.gguf \
  --spec-type draft-mtp --model-draft mtp-gemma-4-12b-it-Q8_0.gguf \
  --spec-draft-ngl 999 --spec-draft-n-max 5 \
  -ngl 999 -c 4096 -fa on -fit off -np 1

⭐ Note that the fastest setting is not the one with the highest draft acceptance. Acceptance measures how often the drafter is right; throughput also pays for the verification work. Sweep n-max and rank on measured t/s, not on acceptance.

⛔ Known defect: long generations abort the server with the draft head enabled

With --spec-type draft-mtp at -c 4096, a 1500-token generation reliably aborts the server:

server-context.cpp:395: GGML_ASSERT(spec_i_batch.empty()) failed
  in server_slot::update_batch(llama_batch&)

Reproduced at both --spec-draft-n-max 3 and 5. 300-token generations are unaffected — we ran 5 per quant here plus 13 consecutive requests in a separate test with no failure, so the decode figures above are sound. We have not yet isolated whether the trigger is the single-request length itself or a context shift near the -c limit; if you enable the drafter, cap generations conservatively and give yourself context headroom. Without --spec-type the model handles long generations normally.

⛔ MTP and vision cannot be used together (upstream llama.cpp PR #20277 — image embeddings are injected outside the token path and the speculative batch loses its boundary). Run two configurations: text with the drafter, images with -fa off and no --spec-type.

A note on measurement, because we hit it here

This model showed a reproducible ~8% dip in a minority of decode samples on an otherwise idle box, while E2B/E4B measured on the same machine and harness were flat. We re-ran at 12 samples per quant to characterise it:

• —Q4_0_ROCMFP4_COHERENT [pyfunc] median 26.93 — 26.94 / 26.93 / 26.93 / 26.93 / 26.93 / 26.94 / 26.94 / 26.93 / 26.93 / 26.93 / 26.95 / 26.94
• —Q4_0_ROCMFP4_COHERENT [story] median 26.93 — 26.92 / 26.93 / 26.93 / 26.93 / 26.93 / 26.92 / 26.93 / 26.94 / 26.92 / 26.93 / 26.93 / 26.92
• —Q4_0_ROCMFP4_FAST_COHERENT [pyfunc] median 26.57 — 26.58 / 26.6 / 26.64 / 26.57 / 26.57 / 26.58 / 26.57 / 26.57 / 26.57 / 26.57 / 26.56 / 26.58

The medians above are taken over 12 samples for this reason. We report every raw sample rather than a summary so you can see the distribution yourself.

Verification

Each artifact was loaded on real hardware and checked for: exact stat bytes vs the dry-run projection, actual token_embd type, three correctness answers, and a 5-sample decode median with two warm-ups discarded. Vision was verified separately with the mmproj on a four-quadrant colour image.

Credits

Base model google/gemma-4-12b-it. BF16 GGUF source from unsloth/gemma-4-12b-it-GGUF. FP4/FP8 tensor types from the ROCmFPX fork of llama.cpp.