KikoCis/gemma-4-31b-it-IQ3_XS-GGUF

Gemma 4 31B IT — IQ3_XS GGUF

Gemma 4 31B Instruct quantized to IQ3_XS (3.40 BPW, 13.1GB) using llama.cpp with importance matrix calibration on CLI/package management data.

Matches f16 baseline on Char-F1 (84.76%) at 1/4.7 the size (50-example NL2Bash benchmark).

### 💡 Looking for the smallest possible variant? Try our [gemma-4-31b-it-IQ2_M-GGUF](https://huggingface.co/KikoCis/gemma-4-31b-it-IQ2_M-GGUF) — 10.17 GB with F1 84.71% and BLEU-4 22.39 (beats f16 at 6× smaller). Custom CLI-tuned imatrix on IQ2_M.

Key Stats

NL2Bash Benchmark — Full Comparison

50 examples from the official Stanford/Tellina test split, reasoning OFF, max_tokens=200, temp=0.1, sorted by Char-F1.

Key observations

• —This IQ3_XS ties f16 on Char-F1 (84.76% vs 84.76%) and comes within 2 F1 of the best available quantization
• —Unsloth Dynamic 2.0 is the Pareto frontier: UD-IQ2M (10.75 GB) is strictly better than plain Q2K (+25.4 F1 points at same BPW) thanks to per-layer adaptive bit allocation and Gemma-4-specific imatrix
• —Q2_K at 2.7 BPW collapses on Gemma 4 (F1 58.6%, produces repetitive output). Standard 2-bit scalar quantization is not viable for this architecture
• —f16 only wins on token-exact metrics (BLEU-4, EM) — IQ3_XS produces semantically equivalent commands with minor stylistic variations (different flag order, quoting style)

Recommendation

• —Smallest working quant: Unsloth UD-IQ2_M (10.75 GB, F1 84%)
• —Best overall: Unsloth UD-IQ3_XXS (11.84 GB, F1 85%)
• —This model (IQ3_XS, 13.1 GB): simpler imatrix calibration (CLI-focused), slightly larger but within 0.3 F1 of UD-IQ3_XXS

Full per-question predictions + layer analysis study: https://huggingface.co/datasets/KikoCis/gemma4-31b-layer-study

CLI Benchmark Results (7/7 with thinking)

Tested with llama-cli -cnv --reasoning on --reasoning-budget 512:

Usage

# llama.cpp with thinking
llama-cli -m gemma4-31b-IQ3_XS.gguf -cnv -ngl 99 --ctx-size 8192 \
  --reasoning on --reasoning-budget 512

# Ollama
cat > Modelfile << 'EOF'
FROM ./gemma4-31b-IQ3_XS.gguf
PARAMETER temperature 0.1
PARAMETER num_ctx 8192
EOF
ollama create gemma4-31b-iq3xs -f Modelfile

Quantization Details

• —Tool: llama.cpp (build 0d049d6)
• —Imatrix: Computed from 200 chunks of synthetic CLI/package management data
• —Source: Converted from google/gemma-4-31b-it safetensors via convert_hf_to_gguf.py
• —Architecture: Gemma4 with sliding + full attention pattern (every 6th layer is full attention)

Hardware Requirements

• —Minimum RAM: 16GB (with partial offload)
• —Recommended: Apple Silicon with 32GB+ unified memory
• —Performance: ~21 tok/s on Apple Silicon (128GB unified) with full GPU offload

Benchmark Methodology

• —NL2Bash test set: 50 examples from the official deterministic split (RANDOM_SEED=100, fold 11 from TellinaTool/nl2bash)
• —Inference: Each question runs as an isolated llama-cli subprocess with -p prompt, --reasoning off, --ctx-size 4096, --max-tokens 200, --temp 0.1
• —Scoring: NLTK BLEU corpus-level + character-level F1 + exact match
• —Reproducibility: All scripts and JSONL predictions are public at the dataset repo above

Real-World Agent Test Warning (April 2026)

Benchmark scores do not predict agent capability. In Docker-based autonomous testing, fine-tuned E4B models (95% BFCL) scored 0/10 while the unfine-tuned base scored 6/10. Fine-tuning for BFCL destroyed general reasoning (error recovery, strategy adaptation, anti-repetition). Fine-tuned E4B models have been withdrawn. For autonomous agent tasks, use the base Gemma 4 model or a larger model at higher BPW. See: The Benchmark Trap — Full Study