Modelpublic

megabytes/gemma-4-12B-it-heretic-volta-gguf

sourceHugging Faceapache-2.0updated 8d agoView on Hugging Face

0likes521downloads

Model Card

megabytes/gemma-4-12B-it-heretic-volta-gguf

This repository contains the complete Gemma 4 12B IT QAT Heretic suite specifically optimized for NVIDIA Volta (Compute Capability 7.0 / Tesla V100 / Titan V) GPUs.

Included Models

File	Size	Format	Description
`gemma-4-12b-it-qat-heretic-volta-Q4_0.gguf`	6.8 GB	`Q4_0` + `Q8_0`	Main LLM with `Q8_0` token embeddings (avoids `Q6_K` stalls on Volta) and canonical chat template
`mmproj-gemma-4-12b-it-qat-heretic-volta-F16.gguf`	117 MB	`F16`	Multimodal vision projector mapped directly to Volta FP16 Tensor Cores
`mtp-gemma-4-12b-it-qat-heretic-volta-Q4_0.gguf`	242 MB	`Q4_0`	Multi-Token Prediction (MTP) speculative decoding assistant model

Optimizations for Volta (sm_70)

1.Avoidance of K-Quants (`Q6_K` → `Q8_0`): Eliminates ALU unpacking bottlenecks on Volta CUDA cores for the 262k-token vocabulary.
2.Native FP16 Vision Projector (`F16`): Runs vision embeddings directly on Volta's FP16 Tensor Cores without integer dequantization overhead.
3.Speculative Decoding Assistant (`Q4_0`): Compressed to 242 MB to maximize speculative drafting throughput.
4.Canonical Delimiters & Chat Template: Fully aligned with Gemma 4 turn delimiters (<|turn>, <turn|>) and Jinja chat templating.

Benchmark Performance (Tesla V100-PCIE-12GB)

Workload	Throughput	Notes
Synthetic Prompt (`pp512`)	270.2 t/s	Raw base model throughput
Synthetic Generation (`tg128`)	50.5 t/s	Raw base model throughput
Text Generation (with MTP drafter)	59.5 t/s	`--spec-draft-n-max 2 --poll 100`
Multimodal Vision Ingestion	94.4 t/s	Native `F16` projector

Recommended llama.cpp / llama-server Command

bash

./llama-server \
  -m ./gemma-4-12b-it-qat-heretic-volta-Q4_0.gguf \
  --mmproj ./mmproj-gemma-4-12b-it-qat-heretic-volta-F16.gguf \
  --model-draft ./mtp-gemma-4-12b-it-qat-heretic-volta-Q4_0.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  --poll 100 \
  -dev CUDA0 \
  -ngl all \
  -devd CUDA0 \
  -ngld all \
  --mmproj-offload \
  --spec-draft-backend-sampling \
  -fa on \
  --jinja