sakamakismile/gemma-4-12B-coder-fable5-composer2.5-MTP-NVFP4

💻 Gemma-4-12B-Coder (fable5 × composer2.5) — NVFP4A16 for vLLM ✨

A faithful 4-bit build of yuxinlu1's coding model, now runnable in vLLM — with a bundled MTP draft for ~1.6× interactive speed. 🚀

TL;DR — A local Python-coding assistant that thinks before it codes. 8.25 GB, runs on one 16 GB Blackwell GPU, native in vLLM (no --quantization flag). Bundled speculative-decode draft included. 💚

🙏 Credit & what this is

This is a weight-only NVFP4 (W4A16) re-quantization of [`yuxinlu1/gemma-4-12B-coder-fable5-composer2.5-v1`](https://huggingface.co/yuxinlu1/gemma-4-12B-coder-fable5-composer2.5-v1-GGUF) — full credit and thanks to [@yuxinlu1](https://huggingface.co/yuxinlu1) for the model and the lovely training recipe. Please ⭐ and follow the original repo; if you want a v2, that's the author's signal to watch.

The author's design intent (preserved here): a focused fine-tune of google/gemma-4-12B-it on verifiable Python coding — distilled from real chain-of-thought (Composer 2.5, kept only where the code passed its tests) plus a Fable 5 "second-attempt" set that recovers the hard cases the main teacher missed. The result reasons in the open (edge cases, complexity) in Gemma's native thinking channel, then emits a clean, runnable solution. It is Python/algorithmic-focused, de-refused (not safety-aligned — add your own guardrails), and English-centric.

Why this build exists: the author shipped GGUF only (great for llama.cpp). This repo reconstructs a vLLM-native artifact so you can serve it with continuous batching, tensor-parallelism, and speculative decoding on Blackwell GPUs.

How it was made (provenance, for the curious): the author's `Q8_0` GGUF (≈lossless) was dequantized to BF16, the gemma-4 language tensors grafted onto a same-arch gemma4_unified skeleton, then quantized to NVFP4A16 with llm-compressor. Quality was verified to match the Q8 source (see below). W4A16 (weights FP4, activations BF16) is used deliberately: the base is non-QAT, where full W4A4 collapses on this architecture — weight-only keeps it robust.

📊 How good is it? (independent eval, greedy pass@1)

Strong at: hard algorithms (DP, graphs, Fenwick/segment trees, bitmask DP), bug-fixing & refactoring (accurate root-cause + genuine O(n²)→O(n) rewrites that preserve semantics), and faithful open reasoning that matches the emitted code. Japanese prompts cause no measurable Python-quality drop.

⚠️ Know the one sharp edge (verified): on quant / time-series code it can write a look-ahead bias (e.g. an unshifted position × a forward-shifted return), and its reasoning sometimes states the correct rule while the code does the opposite. Do not ship its pandas/numpy back-test or accounting code unreviewed — gate it. It's a superb algorithm/debug specialist, not an unsupervised quant author.

🚀 Run it — pick your path

You need: a Blackwell GPU (SM120 / RTX 50-series / RTX PRO / GB10 / B100/200), Docker with the NVIDIA runtime. Gemma-4 unified is new, so you need a vLLM build that registers `Gemma4UnifiedForConditionalGeneration` (recent nightly). vLLM auto-detects the NVFP4 weights — no --quantization flag.

🟢 Easiest — one GPU, just chat (start here)

# download (~8.25 GB)
hf download sakamakismile/gemma-4-12B-coder-fable5-composer2.5-MTP-NVFP4 --local-dir ./model

docker run --rm --gpus '"device=0"' --ipc=host --shm-size 16gb -p 8000:8000 \
  -v $PWD/model:/model:ro \
  vllm/vllm-openai:nightly \
  --model /model --served-model-name gemma4-coder \
  --max-model-len 16384 --gpu-memory-utilization 0.92 --trust-remote-code

Then open the OpenAI-compatible endpoint at http://localhost:8000/v1.

🧠 IMPORTANT — turn the thinking channel ON

This model was trained to think first. In vLLM you must enable it per request (otherwise it skips reasoning and quality drops on hard problems):

curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "gemma4-coder",
  "messages": [{"role":"user","content":"Write a function that returns the longest palindromic substring. Think through edge cases first."}],
  "temperature": 0.0,
  "chat_template_kwargs": {"enable_thinking": true}
}'

In the Python OpenAI client, pass it via extra_body:

from openai import OpenAI
c = OpenAI(base_url="http://localhost:8000/v1", api_key="x")
r = c.chat.completions.create(
    model="gemma4-coder",
    messages=[{"role":"user","content":"...your coding task..."}],
    temperature=0.0,                       # greedy = deterministic code
    extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
print(r.choices[0].message.content)

💡 Sampling: greedy (temperature 0) for deterministic solutions, or the author's temp 1.0, top_p 0.95, top_k 64 for variety (top_k via extra_body).

⚡ Fastest interactive — TP=4 + bundled MTP speculative decode

A 0.4 B MTP draft is bundled in assistant/ (Google's gemma-4-12B-it assistant). It's lossless (the target verifies every token) and gives ~1.6× single-stream speed. Use num_speculative_tokens: 3 (stable optimum) and --kv-cache-dtype fp8 (NVFP4 KV would break the draft):

docker run --rm --gpus '"device=0,1,2,3"' --ipc=host --shm-size 16gb -p 8000:8000 \
  -e NCCL_P2P_DISABLE=1 \
  -v $PWD/model:/model:ro \
  vllm/vllm-openai:nightly \
  --model /model --served-model-name gemma4-coder \
  --tensor-parallel-size 4 --disable-custom-all-reduce \
  --kv-cache-dtype fp8 \
  --speculative-config '{"method":"mtp","model":"/model/assistant","num_speculative_tokens":3}' \
  --max-model-len 16384 --gpu-memory-utilization 0.90 --trust-remote-code

The bundled draft was trained on base gemma-4-12B-it. On this coder fine-tune it stays lossless; acceptance (and thus the exact speedup) may be a touch lower than a coder-native draft. Measured numbers below.

🔌 Multi-GPU without NVLink (consumer / entry Blackwell over PCIe)

There is no working GPU P2P on plain PCIe, so tensor-parallel hangs unless you disable both NCCL P2P and vLLM's custom all-reduce:

  -e NCCL_P2P_DISABLE=1 \          # <-- env; else hangs at NCCL init
  --tensor-parallel-size 4 \
  --disable-custom-all-reduce \    # <-- flag; else the forward deadlocks

Flag cheat-sheet

📈 Throughput (measured — 4× RTX PRO 2000 Blackwell, 16 GB, PCIe / no-NVLink)

Single-stream decode (1 request, 512 tok, thinking on):

Aggregate throughput (no spec-decode; turn MTP off for batch):

Choosing a layout on a fixed GPU budget: TP=4 gives the lowest latency, but TP=2 is more efficient per GPU (≈316 vs 195 tok/s/GPU at 16-way). For max farm throughput, run two data-parallel TP=2 replicas (≈1.3k tok/s on 4 GPUs) instead of one TP=4. Rule of thumb: MTP on for interactive (≤8 concurrent), off for high-concurrency batch.

🔧 Quantization details

📚 Base, license, and a note on use

• —Original model: `yuxinlu1/gemma-4-12B-coder-fable5-composer2.5-v1` (fine-tune of `google/gemma-4-12B-it`).
• —MTP draft: `google/gemma-4-12B-it-assistant`, bundled in assistant/.
• —License: Gemma Terms of Use — derivatives must comply.
• —De-refused / not safety-aligned: add your own guardrails for production. Strongest on Python / algorithmic tasks; double-check general facts and especially time-series / quant code. Shared as-is, no warranty. Happy hacking! 🐾✨

NVFP4 build & eval by [Lna-Lab](https://huggingface.co/sakamakismile). Thanks again to @yuxinlu1 for the original.