axiom-of-choice/gemma-3-4b-es-reasoning-gguf

gemma-3-4b-es-reasoning-gguf

GGUF quantizations for llama.cpp, Ollama and LM Studio. Quantized from a local merge of `axiom-of-choice/gemma-3-4b-es-reasoning-peft` onto google/gemma-3-4b-it-qat-q4_0-unquantized.

Text-only. google/gemma-3-4b-it-qat-q4_0-unquantized is multimodal; this GGUF is not -- the vision tower was dropped during conversion (convert_hf_to_gguf.py converts only the text decoder for this architecture; verified with gguf.GGUFReader: 0 vision tensors in the file). Defensible for a Spanish-reasoning text model, but this repo cannot see images even though the base it was merged onto can -- use the bf16 merge, or the source adapter on its multimodal base, if you need that.

Files

• —Q4_0
• —Q8_0

Shipped as `Q4_0`, not the usual `Q4_K_M` default. This base is QAT (quantization-aware training) for exactly the q4_0 scheme -- it's in the base model's own name. Measured directly: Q4_K_M silently produced wrong arithmetic and dropped the Spanish <think> reasoning block that this adapter exists to add, on prompts where Q8_0 (near-lossless) and Q4_0 both answered correctly with full reasoning. Not a hypothesis -- verified with llama-cli on the same question across all three quant levels before publishing. If you need a smaller/different quant than these two, test it against <think> presence and correctness before trusting it; K-quants are not verified safe for this model.

Q4_0 is the smallest with negligible quality loss for most use. Q8_0, where offered, is near-lossless if you have the RAM/VRAM for it.

Usage

ollama run hf.co/axiom-of-choice/gemma-3-4b-es-reasoning-gguf:Q4_0
# or
llama-cli -hf axiom-of-choice/gemma-3-4b-es-reasoning-gguf:Q4_0 -p "¿Cuánto es 17 por 24?"

What the numbers below do NOT cover

The merge-verification table is measured on the bf16 merge, before this GGUF's own quantization step. Q4/Q8 quantization is a further lossy step on top of that and is not separately measured here -- treat it as an additional, unquantified source of drift on top of the number below, not as covered by it.

Did merging (pre-quantization) change the model?

This is checked against the unmerged PEFT adapter's own generations (results/peft_parity/gemma-3-4b-s5.6-qat-unquant.json), not the original MLX numbers -- the question here is only "did merging change the model", and MLX-vs-PyTorch divergence is a separate, already-documented story on the adapter repo's card.

Paired counts: 0 correct only here, 0 correct only unmerged, out of 20. The merge tracks the unmerged PEFT adapter closely, as it should -- this is the same engine and precision on both sides, so unlike the MLX-vs-PyTorch phase-2 numbers, there is no expected source of divergence here. Treat any drift beyond this as a merge bug, not backend noise.

Training

Same recipe, dataset and evaluation as the adapter -- see `axiom-of-choice/gemma-3-4b-es-reasoning-peft`.