Modelpublic

tiyuvta/gemma-4-31B-it-assistant-MTP-GGUF

sourceHugging Faceapache-2.0updated 15d agoView on Hugging Face

1likes484downloads

Model Card

gemma-4-31B-it-assistant — MTP drafter GGUF (Q8_0 + NVFP4)

GGUF builds of google/gemma-4-31B-it-assistant, the official 4-block assistant (MTP) drafter for google/gemma-4-31B-it, quantized for speculative decoding of gemma-4-31B-it GGUF trunks.

The assistant is a small draft head (4 layers, hidden 1024, tied embeddings, 262k vocab) that computes only its own queries and reads the target model's KV cache (attention_k_eq_v, all 4 layers KV-shared), so its per-draft-token cost is a small fraction of a target decode step. The target model verifies every drafted token; the emitted stream is the target's own output.

Files

file	quant	bytes	sha256
`gemma-4-31B-it-official-Q8_0-MTP.gguf`	Q8_0	514,666,944	`f5a8758752b1195623a7e22f185b4445d61f19086eaaf9125b47c631ef7a9b66`
`gemma-4-31B-it-official-NVFP4-MTP.gguf`	NVFP4 (6.79 BPW; norms/embeddings kept high-precision)	414,134,720	`f3374652c1d302cbcd2d72480042965233dea493b72e97d8c14a1dd223844d24`

Both files carry the full 17,336-byte gemma-4 chat template in their GGUF metadata, byte-identical to the source checkpoint's (verified on the exact published bytes — quantization tooling can silently drop chat templates, so this was gated, not assumed).

How they were made

—The official google/gemma-4-31B-it-assistant bf16 checkpoint was converted to an F16 GGUF with a byte-parity gate against the source weights (per-layer scalars byte-equal; output norm exact on all 1024 rows).
—llama-quantize produced the Q80 and NVFP4 files from that F16 conversion (no imatrix). NVFP4 is a Blackwell-native 4-bit float format; it was measured on RTX PRO 6000 Blackwell (sm120). On other hardware, use the Q8_0 file.
—The F16 conversion itself is deliberately not published as a drafter: a float MTP head does not arm the tiyuvta serving engine's gemma speculative route; the quantized builds are the servable artifacts.

Measured results

All numbers are our own measurements of these exact files. Acceptance is protocol-dependent — trunk quantization, sampling, workload shape and verification policy all move it — so every number below states its protocol.

Acceptance / identity / throughput A/B (fixed workload)

Protocol: the serving engine (v0.95.0/v0.96.0 candidates), greedy decoding, single stream, 1× RTX PRO 6000 Blackwell 96 GB; trunk = a Q6_K-class GGUF build of gemma-4-31B-it; draft depth K=5; 12 held-out agent-shaped prompts; interleaved ×5 with one fresh boot per arm per repetition; correctness gate outranks speed — the speculative stream must reproduce plain greedy decode byte-exactly.

arm	acceptance (accepted/drafted)	tok/s median	byte-identity vs plain
Q8_0	67/135 = 0.496, bit-reproducible every repetition	118.0	30/30
NVFP4	67/135 = 0.496, numerically identical to Q8_0 every repetition	118.3	30/30
plain reference (same cell, drafter attached, spec route off)	—	58.2	—

The two quantizations are interchangeable on this protocol: acceptance is identical to the count, outputs are byte-identical to plain decode in both arms, and throughput is a wash (~2.03× plain on this cell). What NVFP4 buys is ~100.5 MB on disk/transfer and ~96 MiB resident VRAM.

Hosted-endpoint A/B (production serving config, measured 2026-08-20)

Four fixed greedy probes (256-token budget) through our then-hosted gemma-4-31B-it endpoint, minutes apart, Q80 vs NVFP4 under the identical serving configuration: outputs **byte-identical on 4/4 probes** (content and reasoning channels); probe-set acceptance 549/817 = 0.672 (Q80) vs 547/823 = 0.665 (NVFP4) — a wash.

Both files served that endpoint in production: the Q8_0 build until 2026-08-20, then the NVFP4 build until the hosted gemma-4-31B-it endpoint was retired on 2026-08-21. The measurements above are dated production receipts, not claims about a live service.

Lineage pairing matters

This is the official-lineage head (minted from the bf16 gemma-4-31B-it-assistant checkpoint). In our A/Bs it measured 0.58–0.60 prose acceptance on a trunk built from the official gemma-4-31B-it weights, and 0.28–0.34 when cross-paired with a QAT-lineage trunk. Pair this drafter with trunks derived from the official weights; QAT trunks want the QAT assistant head.

Serving

The serving engine serves gemma-4-31B-it with this drafter attached, and the gemma speculative route arms automatically (K=5; engages on greedy, unconstrained, text-only sessions; speculative output is gated byte-identical to plain decode).

Hosted inference

The hosted gemma-4-31B-it endpoint this drafter served was retired on 2026-08-21 (dated receipts above). The same lab runs a production inference API at api.tiyuvta.ai — currently serving Qwen3.8 27B at native 262,144-token context (OpenAI Chat Completions, Responses, and Anthropic Messages on one endpoint, tool calling included), on the same serving engine and exactness gates used for every number on this card. Docs: inference.tiyuvta.ai/docs.

License

apache-2.0, inherited from google/gemma-4-31B-it-assistant; see the Gemma 4 license note linked from the base model card.