fraQtl/Llama-3.1-8B-Instruct-Hi-Fi-GGUF

Llama-3.1-8B-Instruct Q4KM GGUF — fraQtl Hi-Fi (calibration-aware quantization)

Built with Llama.

Iso-size Q4KM-class GGUF of meta-llama/Llama-3.1-8B-Instruct built with fraQtl calibration-aware per-tensor quantization: same file-size class as the leading community Q4KM (within 0.007%), measurably closer to the original bf16 model's output distribution on both locked evaluation slices.

This is the target half of the fraQtl Llama Draft Pair (its speculative-decoding draft is `fraQtl/Llama-3.2-1B-Instruct-Hi-Fi-GGUF`); the pair receipt ships in the collection when its benches complete.

Evaluation — fidelity to the original model

Metric: symmetric top-20 KLD on the support of a Q80 teacher **quantized from the original pinned bf16 checkpoint** (this measures fidelity-to-original — the axis a user migrating from the bf16 model experiences). 3 independent runs, run-to-run drift 0.0 (deterministic rig). Baseline credited: [bartowski](https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF) Q4K_M — the canonical community quantization (pinned revision bf5b95e9…).

Slices: code/math 400-record held-out (sha cce68602…b1665) and general chat/tool/wikitext 400-record (sha b10a79ca…276). Eval runtime: llama-cpp-python==0.3.33 (vendored llama.cpp 78d2f5246), A100-80GB, seed 0.

Long-context retrieval (needle-in-a-haystack)

Grid: contexts {8192, 32768} × depths {0.1, 0.5, 0.9} × 3 passkeys (18 cells), exact-match, greedy, 3 runs per arm. Both this artifact and the bartowski baseline score 18/18 on every run — the grid saturates at 8B scale; we state parity, not a win. Note: greedy decoding, but CUDA reduction order varies across instances, so retrieval scores are always reported as multi-run.

Provenance

Architecture / allocation

Standard dense Llama (32 layers, GQA). The type placement is chosen per tensor by calibration-aware allocation at the identical total byte budget as the baseline — the claim is smarter bits, not more bits. Attention V is the most protected family (Q80 on 29/32 layers); attention Q is never demoted below the baseline's native type. The lmhead stays at the baseline's Q6_K.

Usage

# llama.cpp
llama-cli -m Llama-3.1-8B-Instruct-fraQtl-HiFi-Q4_K_M.gguf -ngl 99 -c 8192 \
  -p "your prompt" --temp 0.7

# speculative decoding with the fraQtl 1B draft (pair receipt to follow)
llama-speculative -m Llama-3.1-8B-Instruct-fraQtl-HiFi-Q4_K_M.gguf \
  -md Llama-3.2-1B-Instruct-fraQtl-HiFi-Q4_K_M.gguf \
  --draft-max 16 -ngl 99 -ngld 99 -p "your prompt"

The GGUF embeds the chat template; use llama_chat_apply_template / chat-completion APIs.

fraQtl ladder

• —Hi-Fi GGUFs (this repo + the 1B draft) — download-and-run quality tier
• —Qwen3-4B KV sidecars — the serving lane (long-context KV compression receipts)

License

Llama 3.1 Community License (full text in LICENSE; attribution in NOTICE). Derivative model name begins with "Llama" per the license. Use is subject to Meta's Acceptable Use Policy referenced therein.

Limitations

• —Fidelity numbers measure distributional closeness to the original model, not task accuracy; no task-benchmark claims are made on this card.
• —Q4KM-class quantization: outputs can differ from the bf16 model.
• —The retrieval grid saturates for both arms at 8B — it differentiates the 1B tier, not this one.

The fraQtl ladder

Same discipline at every tier: pinned provenance, measured numbers, losses disclosed.

More from fraQtl

The serving lane — KV-cache compression sidecars for vLLM — holds nine concurrent ≈128K-context users on a single A100 (134.1 tok/s aggregate, 9/9 per-user retrieval checks, receipt 2026-08-14): fraQtl/qwen3-4b-instruct-2507-kv-sidecars. Org page: huggingface.co/fraQtl.