yuxinlu1/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF

✨ Gemma4-12B-Reasoning-Distill (GGUF) ✨

🐣 Tiny footprint, big brain — local AI for everyone

No matter your GPU. No matter your RAM. If you've got ~4.5 GB of VRAM or unified memory free, you can run your own private, offline AI right now. 🚀 Tuned on Opus 4.6, 4.7 & 4.8 reasoning data, it delivers a major leap in reasoning power — whether you're asking questions or writing code. 🧠💻 All local, all yours, no API, no cloud.

⚡ NEW — the MTP version is here! Free speed 🎉

As of June 7, 2026, mainline llama.cpp just merged Gemma 4 MTP support — so the MTP draft model is now live in the `MTP/` folder. Drop it next to any quant and generation gets noticeably faster with identical output (speculative decoding is lossless) — just add a couple of flags. 👉 See ⚡ Speed it up with MTP below. 💚

📦 Pick your size (GGUF quants)

🧮 "Will it fit?" — context length cheat-sheet

Rough estimates 🤓 (assumes q8_0 KV cache + ~1.5 GB overhead; use `q4_0` KV cache for ≈2× more context!). Max context is 131K. "—" = won't fit, pick a smaller quant. ✂️

💡 Apple Silicon / integrated GPUs with unified memory count too — same numbers, just slower than a dGPU. 💡 Low on room? Drop a quant or switch KV cache to q4_0 and your context roughly doubles.

⚡ Speed it up with MTP (free & lossless) 🏎️

New as of June 7, 2026! Gemma 4's Multi-Token Prediction drafter lets the model guess a few tokens ahead and verify them in one shot — so you get more tokens/sec with byte-for-byte identical output. Pure speed, zero quality cost. 🪄

1. Grab the tiny draft from the `MTP/` folder:

💡 The draft is tiny — keep it Q8 or higher (over-quantizing a draft just lowers its hit rate). It pairs with any quant of the main model.

2. You need a fresh llama.cpp build — June 7 2026 (`b9553`) or newer. MTP was just merged, so older builds can't load the draft (unknown architecture: 'gemma4-assistant').

3. Run it exactly like below, just +3 flags (--model-draft, --spec-type, --n-gpu-layers-draft):

@echo off
cd /d C:\llama.cpp
llama-server.exe ^
  -m C:\models\gemma4-opus48-Q4_K_M.gguf ^
  --model-draft C:\models\MTP\gemma-4-12B-it-MTP-Q8_0.gguf ^
  --spec-type draft-mtp --spec-draft-n-max 4 ^
  --ctx-size 16384 --n-gpu-layers 99 --n-gpu-layers-draft 99 ^
  --no-mmap -fa on ^
  --temp 1.0 --top-p 0.95 --top-k 64 ^
  --host 0.0.0.0 --port 18080
pause

Measured on a single RTX 5090 (Q4KM main + Q8 draft): ~1.3× faster at greedy and ~1.2× at the default thinking sampling — free, with no change to output. 🎈

🔧 Heads-up: this is the stock Gemma drafter (trained on base Gemma 4), so on this fine-tune the hit rate — and thus the speedup — is a little lower than on vanilla Gemma 4. A re-aligned draft could push it higher (maybe a future update). Either way: free speed, no downside. 💚

🚀 How to run it (super easy)

Option A — llama.cpp (recommended) 🦙

1. 1.Grab a quant above (e.g. …-Q4_K_M.gguf) and llama-server from llama.cpp.

⚠️ Needs a recent llama.cpp (this is the gemma4_unified architecture — older builds won't load it).

1. 1.Run a server (Windows .bat shown — tweak --port, --ctx-size to taste):

@echo off
cd /d C:\llama.cpp
llama-server.exe ^
  -m C:\models\gemma4-opus48-Q4_K_M.gguf ^
  --ctx-size 16384 ^
  --n-gpu-layers 99 ^
  --no-mmap ^
  -fa on ^
  --cache-type-k q8_0 --cache-type-v q8_0 ^
  --temp 1.0 --top-p 0.95 --top-k 64 ^
  --host 0.0.0.0 --port 18080
pause

1. 1.Open http://localhost:18080 and chat. 🎉 (Tip: bump --ctx-size per the table; use q4_0 KV for more.)

Option B — one-click apps 🖱️

Works in LM Studio, Jan, Ollama, etc. — just import the GGUF, pick your quant, go. 🐾

🧠 Thinking mode

This model thinks in Gemma's native thought channel. Keep `enable_thinking=true` (the default chat template handles it). Recommended sampling: temp 1.0, top_p 0.95, top_k 64.

⚠️ Good to know

• —Reduced refusals: the training data omits safety hedging, so this refuses less than the base model. It is not safety-aligned — add your own guardrails for production. Use responsibly. 🙏
• —Reasoning is stylistic synthetic CoT — great for structure, but double-check facts/numbers.
• —English-centric.

📚 Data & License

• —License: Apache 2.0. Gemma 4 is released by Google under [Apache 2.0](https://ai.google.dev/gemma/apache_2) (unlike the older Gemma 1/2/3 terms), so this derivative is Apache 2.0 as well.
• —Base model: `google/gemma-4-12B-it`.
• —Training data: built on the public, Apache-2.0 dataset `angrygiraffe/claude-opus-4.6-4.7-reasoning-8.7k`, augmented with additional Opus 4.8-generated reasoning samples I curated and mixed in.
• —Personal/hobby project — shared as-is, no warranty. Have fun! 🐾✨