CoolFace
Modelpublic

Zhongzhu/OSCAR-LLAMACPP-Gemma-4-12B-it-INT2-KV

sourceHugging Faceupdated 4mo agoView on Hugging Face
0likes130downloads
Model Card

Gemma 4 12B IT — OSCAR INT2 KV cache (GGUF)

Gemma-4-12B-it in GGUF, packaged for the OSCAR ~2-bit (INT2) KV cache fork of llama.cpp. Two independent axes are combined here:

  • —weights: bf16 (full precision) or Q4_K_M (~4.5-bit) — affects model size/quality, not the KV cache.
  • —OSCAR rotation: rot-kv = the per-layer calibrated rotation is baked into the GGUF, which is what makes the 2-bit KV cache keep near-f16 quality. base = plain model, no rotation.

The -rot-kv models need the OSCAR fork (-fa on + --cache-type-k/v q2_0 + the env vars below). The base models are standard GGUFs that run on stock llama.cpp with a normal f16 KV cache.

Folders

folderweightsOSCAR rotationsizewhat it's for
`q4km-rot-kv/` ⭐Q4KM✅ baked in6.9 GBRecommended. Small weights + OSCAR INT2 KV — the practical deployment model.
bf16-rot-kv/bf16✅ baked in22 GBFull-precision weights + OSCAR INT2 KV. Use to isolate the KV-cache effect (identical weights, compare f16-KV vs INT2-KV) or for max quality.
base-q4km/Q4KM❌ none6.9 GBPlain Gemma 4 12B, Q4KM weights. Runs on stock llama.cpp (f16 KV).
base-bf16/bf16❌ none22 GBPlain Gemma 4 12B, full precision. Stock llama.cpp, or use as the base to bake your own rotation.
rotation/—the matrices2×18 MBRaw OSCAR calibrated rotation (k_/v_rotation_*.pt, per-layer 512×512). Bake onto any Gemma 4 12B base GGUF.
"OSCAR rotation" is an in-graph, post-RoPE orthogonal transform applied to Q/K (and V) so that the KV cache quantizes well at 2-bit. The -rot-kv GGUFs already contain it as blk.{i}.attn_{k,v}_rot.weight tensors; the base weights are copied through unchanged.

Run (the -rot-kv models)

Needs the OSCAR fork built with Metal. Gemma 4 also needs its chat template.

bash
LLAMA_KV_FUSED_FA=1 LLAMA_KV_NO_HADAMARD=1 LLAMA_KV_CLIP_RATIO=0.96 \
LLAMA_KV_HP_SINK=512 LLAMA_KV_HP_RECENT=2048 \
./build/bin/llama-server -m q4km-rot-kv/gemma-4-12b-it-rot-kv.gguf \
  -fa on -ngl 99 -c 16384 \
  --cache-type-k q2_0 --cache-type-v q2_0 \
  --chat-template-file models/templates/google-gemma-4-31B-it.jinja \
  --host 127.0.0.1 --port 8080
env varmeaningvalue
LLAMA_KV_FUSED_FAfused INT2+f16 flash-attention kernels (fast Metal path)1
LLAMA_KV_NO_HADAMARDrotation is in-graph, so skip the in-quant Hadamard1
LLAMA_KV_CLIP_RATIOper-row outlier clip before quant0.96
LLAMA_KV_HP_SINK / LLAMA_KV_HP_RECENTtokens kept high-precision (first/last)512 / 2048

--cache-type-v f16 keeps V high-precision (a bit more quality, more memory). A base (non-rotated) model with INT2 flags falls back to degraded data-free INT2 — use a -rot-kv model for INT2.

Bake your own rotation

bash
python3 oscar-rotation/export_rot_kv_gguf.py \
  --base    base-bf16/gemma-4-12b-it-bf16.gguf \
  --rot-dir rotation/ \
  --out     gemma-4-12b-it-bf16-rot-kv.gguf