CoolFace
Modelpublic

Itopoly/G9v3-3B-Q3_K_M-GGUF

sourceHugging Faceapache-2.0updated 23d agoView on Hugging Face
0likes71downloads
Model Card

G9v3-3B Q3KM GGUF — 4-core CPU tier

A CPU-tier-validated Q3KM cut of [ai9stars/G9v3-3B](https://huggingface.co/ai9stars/G9v3-3B) (dense ~3B, LlamaForCausalLM, 131K context, think/no-think modes, XML tool calling), packaged for llama.cpp on low-core machines. The GGUF itself is quantized by [mradermacher/G9v3-3B-GGUF](https://huggingface.co/mradermacher/G9v3-3B-GGUF); this repo adds the serving template and measured performance numbers for the 4-core tier.

What's in this repo

FileSizeWhat it is
g9v3-3B.Q3_K_M.gguf1.5 GBthe quantized model
g9v3_chat_template_low.jinja12 KBchat template — required for tool calling (see below)

Measured performance (4 threads, x86-64 AVX2)

Measured on an 8-vCPU EPYC @ 2.0 GHz run at 4 threads to simulate a 4-core box, with llama.cpp llama-server:

  • —Decode: ~25 tok/s (think-mode ~14 tok/s) — comfortably above the ~10 tok/s interactive floor for a reasoning model
  • —Prompt processing: ~55 tok/s — a 5.5K-token prompt costs ~100 s once; the prefix cache makes repeat requests ~0.6 s
  • —Sanity check: reasoning is separated from the answer, tool calls emit structured tool_calls, and legacy OpenAI function_call history is normalized by the template

RAM need: ≥6 GB. Context memory: KV cache is 52 KiB/token on this model — 8192 ctx fits an 8 GB box, 32768 ctx fits 16 GB.

Run with llama.cpp

bash
# get the files
huggingface-cli download itopoly/G9v3-3B-Q3_K_M-GGUF g9v3-3B.Q3_K_M.gguf --local-dir .
huggingface-cli download itopoly/G9v3-3B-Q3_K_M-GGUF g9v3_chat_template_low.jinja --local-dir .

# serve (OpenAI-compatible on /v1/chat/completions, model name: g9v3-3b)
llama-server -m g9v3-3B.Q3_K_M.gguf --alias g9v3-3b \
  -t 4 -tb 4 -c 8192 --port 8000 --host 0.0.0.0 \
  --chat-template-file g9v3_chat_template_low.jinja

Reasoning text arrives in message.reasoning, the answer in content.

Why the chat template matters

Don't serve this GGUF without g9v3_chat_template_low.jinja: the stock template drops tool-result messages, and the model re-calls the same tool forever (an observed loop bug). The bundled template normalizes legacy tool history so multi-turn tool use works.

Notes

  • —No-think mode: temperature=0.7, top_p=0.95; think mode: temperature=1.0, top_p=0.95.
  • —Prefill is the bottleneck on CPU — keep prompts ≤ 1–2K tokens, or accept a one-time ~2 min first hit per distinct large prefix (cached afterwards).
  • —Tool calling works (verified single-turn + both tool-history formats), but tool-call quality on a 3B is tier-limited; the 39B family models are the tool-heavy choice.

Credits