Modelpublic

fraQtl/Llama-3.2-1B-Instruct-Hi-Fi-GGUF

sourceHugging Facellama3.2updated 1mo agoView on Hugging Face

0likes40downloads

Model Card

Llama-3.2-1B-Instruct Q4KM GGUF — fraQtl Hi-Fi (calibration-aware quantization)

Built with Llama.

Iso-size Q4KM-class GGUF of meta-llama/Llama-3.2-1B-Instruct built with fraQtl calibration-aware per-tensor quantization. Same file-size class as the leading community Q4KM (within 0.02%). Wins the general slice by a wide margin and top-1 agreement on both slices; on the code/math KLD slice it is 0.11% behind the baseline — stated plainly below, both directions.

This is the draft half of the fraQtl Llama Draft Pair (its target is `fraQtl/Llama-3.1-8B-Instruct-Hi-Fi-GGUF`); the speculative-decoding pair receipt ships in the collection when its benches complete.

Evaluation — fidelity to the original model

Metric: symmetric top-20 KLD on the support of a Q80 teacher **quantized from the original pinned bf16 checkpoint** (fidelity-to-original — the axis a user migrating from the bf16 model experiences). 3 independent runs, run-to-run drift 0.0 (deterministic rig). Baseline credited: [bartowski](https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF) Q4K_M — the canonical community quantization (pinned revision 067b946c…).

Arm	bytes	true bpw	KLD code/math ↓	KLD general ↓	top-1 agree cm	top-1 agree gen
bartowski Q4KM	807,694,464	5.2286	0.046045	0.036032	0.9369	0.9227
fraQtl Hi-Fi (this repo)	807,513,952 (−0.02%)	5.2274	0.046093 (+0.11% — behind baseline)	0.031523 (−12.5%)	0.9416	0.9297

Read the table exactly as printed: this artifact loses the code/math KLD comparison by 0.11% relative and wins the general KLD comparison by 12.5% relative; it wins top-1 agreement on both slices. Slices: code/math 400-record held-out (sha cce68602…b1665), general chat/tool/wikitext 400-record (sha b10a79ca…276). Eval runtime: llama-cpp-python==0.3.33 (vendored llama.cpp 78d2f5246), A100-80GB, seed 0.

Long-context retrieval (needle-in-a-haystack)

Grid: contexts {8192, 32768} × depths {0.1, 0.5, 0.9} × 3 passkeys (18 cells), exact-match, greedy. Greedy decoding still shows ±1-cell flicker across container instances (CUDA reduction order), so scores are multi-run ranges, never single-run points:

Arm	18-cell score (4 runs ours / 3 runs baseline)	32K subgrid
fraQtl Hi-Fi	17–18 / 18	9/9 every run
bartowski Q4KM	16–17 / 18	9/9 every run

This artifact never scored below the baseline in any paired run.

Provenance

Field	Value
Base model	`meta-llama/Llama-3.2-1B-Instruct` @ `9213176726f574b556790deb65791e0c5aa438b6`
Conversion + quantization	llama.cpp @ `78d2f524682d9fee790a6460c93d018dafeb5229`; `convert_hf_to_gguf.py --outtype bf16`, then `llama-quantize` with per-tensor `--tensor-type` overrides
Calibration	fraQtl mixed code+math / general packet (sha256 `c88e7e0a…2dabc`), llama-imatrix ctx 2048, BF16 substrate; `imatrix.dat` included
Teacher	Q8_0 quantized from the original bf16 at the same llama.cpp commit
Artifact sha256	`241b1e4848762cc77f2ebee9090e6a898f8541f421f7d854509f5de3a51452bb`
Receipts	`receipts/` in this repo: build + evaluation + retrieval JSONs, per-layer true-bpw table

Architecture / allocation

Standard dense Llama (16 layers, GQA, tied embeddings). Type placement chosen per tensor by calibration-aware allocation at the identical total byte budget as the baseline — smarter bits, not more bits. Attention V is the most protected family (Q80 on 12/16 layers); attention Q is never demoted below the baseline's native type; the tied token-embedding/output table stays at the baseline's Q6K.

Usage

bash

# standalone
llama-cli -m Llama-3.2-1B-Instruct-fraQtl-HiFi-Q4_K_M.gguf -ngl 99 -c 8192 \
  -p "your prompt" --temp 0.7

# as the speculative-decoding draft for the fraQtl 8B (pair receipt to follow)
llama-speculative -m Llama-3.1-8B-Instruct-fraQtl-HiFi-Q4_K_M.gguf \
  -md Llama-3.2-1B-Instruct-fraQtl-HiFi-Q4_K_M.gguf \
  --draft-max 16 -ngl 99 -ngld 99 -p "your prompt"

The GGUF embeds the chat template; use llama_chat_apply_template / chat-completion APIs.

fraQtl ladder

—Hi-Fi GGUFs (this repo + the 8B target) — download-and-run quality tier
—Qwen3-4B KV sidecars — the serving lane (long-context KV compression receipts)

License

Llama 3.2 Community License (full text in LICENSE.txt, included verbatim; attribution in NOTICE). Derivative model name begins with "Llama" per the license. Use is subject to Meta's Acceptable Use Policy referenced therein. Note: the pinned license text distributed with the text-only 1B model contains no European-Union-specific restriction clause; Meta's EU-restriction language in the Llama 3.2 family applies to the multimodal models. The license file in this repo is the unmodified text distributed with the pinned base revision.

Limitations

—Fidelity numbers measure distributional closeness to the original model, not task accuracy; no task-benchmark claims are made on this card.
—The code/math KLD slice is 0.11% behind the credited baseline (see table).
—Q4KM-class quantization of a 1B model: expect the quality limits of the 1B tier; this artifact passed all greedy coherence smokes in our gates (receipts in-repo).

The fraQtl ladder

Tier	Job	Example
Hi-Fi	maximum fidelity at standard 4-bit size	this repo's Q4KM
Hi-Fi Phone	fits phone/laptop RAM, fidelity-first	E2B Phone, 2.86 GB
SmartEdge	smallest footprint that survives	E2B IQ3XXS, 2.45 GB

Same discipline at every tier: pinned provenance, measured numbers, losses disclosed.

More from fraQtl

The serving lane — KV-cache compression sidecars for vLLM — holds nine concurrent ≈128K-context users on a single A100 (134.1 tok/s aggregate, 9/9 per-user retrieval checks, receipt 2026-08-14): fraQtl/qwen3-4b-instruct-2507-kv-sidecars. Org page: huggingface.co/fraQtl.