Modelpublic

Zhongzhu/OSCAR-LLAMACPP-Qwen3-32B-INT2-KV

sourceHugging Faceupdated 4mo agoView on Hugging Face

0likes20downloads

Model Card

Qwen3 32B - OSCAR INT2 KV cache (GGUF)

Qwen3-32B in GGUF, packaged for the OSCAR ~2-bit (INT2) KV cache fork of llama.cpp.

This repository contains a rotation-baked GGUF plus the raw calibrated rotation matrices. The model is intended for OSCAR INT2 KV-cache inference: the per-layer rotation is already embedded in the GGUF, so the KV cache can be quantized to 2 bits while retaining near-f16 quality.

Files

file	contents	size	what it is for
`qwen3-32b-rot-kv.gguf`	Qwen3 32B GGUF with OSCAR rotation baked in	18 GB	Main model for OSCAR INT2 KV-cache inference.
`k_rotation_qqt_r_h_pbr.pt`	raw calibrated K rotation	4.1 MB	Use with a compatible base GGUF if you want to bake the rotation yourself.
`v_rotation_sst_r_h_pbr.pt`	raw calibrated V rotation	4.1 MB	Use with a compatible base GGUF if you want to bake the rotation yourself.

"OSCAR rotation" is an in-graph, post-RoPE orthogonal transform applied to K and V so that the KV cache quantizes well at 2-bit. The -rot-kv GGUF already contains the rotation tensors; no runtime rotation file is needed for normal inference.

Run

Use the OSCAR llama.cpp fork built with the backend you need (for example Metal on Apple Silicon).

bash

LLAMA_KV_FUSED_FA=1 LLAMA_KV_NO_HADAMARD=1 LLAMA_KV_CLIP_RATIO=0.96 \
LLAMA_KV_HP_SINK=512 LLAMA_KV_HP_RECENT=2048 \
./build/bin/llama-server -m qwen3-32b-rot-kv.gguf \
  -fa on -ngl 99 -c 32768 \
  --cache-type-k q2_0 --cache-type-v q2_0 \
  --host 127.0.0.1 --port 8080

env var	meaning	value
`LLAMA_KV_FUSED_FA`	fused INT2+f16 flash-attention kernels, fast path when available	`1`
`LLAMA_KV_NO_HADAMARD`	rotation is in-graph, so skip the in-quant Hadamard	`1`
`LLAMA_KV_CLIP_RATIO`	per-row outlier clip before quant	`0.96`
`LLAMA_KV_HP_SINK` / `LLAMA_KV_HP_RECENT`	tokens kept high-precision (first/last)	`512` / `2048`

--cache-type-v f16 keeps V high-precision, which may improve quality at the cost of more memory.

Bake your own rotation

If you have a compatible base Qwen3-32B GGUF, you can bake the included rotation matrices into it with the OSCAR export script:

bash

python3 oscar-rotation/export_rot_kv_gguf.py \
  --base    qwen3-32b-base.gguf \
  --rot-dir . \
  --out     qwen3-32b-rot-kv.gguf