Datasetpublic

CyberNative-AI/gguf-repro-harness

Two-run reproducibility of a GGUF quantization and evaluation harness: quality and memory reproduce, wall-clock latency does not We ran the same pinned pipeline twice — two different operators, fresh containers, same commands — on two Apache-2.0 models (Qwen2.5-0.5B-Instruct and SmolLM2-360M-Instruct) to measure what a Q4_K_M quantization changes and whether those measurements reproduce. What reproduced across both runs (exact, or within ±2%): the F16 and Q4_K_M GGUF files are… See the full description on the dataset page: https://huggingface.co/datasets/CyberNative-AI/gguf-repro-harness.

sourceHugging Faceapache-2.0updated 3d agoView on Hugging Face

1likes151downloads

Dataset Card

Two-run reproducibility of a GGUF quantization and evaluation harness: quality and memory reproduce, wall-clock latency does not

We ran the same pinned pipeline twice — two different operators, fresh containers, same commands — on two Apache-2.0 models (Qwen2.5-0.5B-Instruct and SmolLM2-360M-Instruct) to measure what a Q4KM quantization changes and whether those measurements reproduce.

What reproduced across both runs (exact, or within ±2%):

—the F16 and Q4KM GGUF files are byte-identical (SHA-256);
—perplexity on the Wikitext-2 test set (fixed 128 x 512-token budget) is identical to four decimals for each arm between runs;
—HellaSwag acc_norm over a fixed 200-task subset is identical between runs for each arm (the arms are not expected to match each other; SmolLM2 F16 52.50% vs Q4KM 54.50%);
—peak RSS during the perplexity run differs by at most 0.13%.

What did not reproduce within ±2%: wall-clock model load time and prompt/generation throughput on our shared CPU host (up to 61% and 34% spread respectively). We therefore make no latency claims from this harness; reproducible latency measurement needs exclusive CPU.

Measured Q4KM vs F16 deltas (run 1 of 2):

model	PPL F16	PPL Q4_K_M	HellaSwag F16	HellaSwag Q4_K_M	peak RSS during the perplexity run F16	peak RSS during the perplexity run Q4_K_M
Qwen2.5-0.5B-Instruct	15.3782	15.7842	50.50%	50.50%	1942.6 MiB	1402.0 MiB
SmolLM2-360M-Instruct	14.2338	14.6453	52.50%	54.50%	1114.1 MiB	702.8 MiB

Harness, exact commands, raw logs and the two-run diff: https://huggingface.co/datasets/CyberNative-AI/gguf-repro-harness. A summary write-up of both runs: https://cybernative.ai/labs/gguf-repro-harness/ Compute: CPU-only, no GPU, $0 new spend. Perplexity is comparable only within a model, between arms. Same-harness repeats are repeat execution, not independent scientific reproduction.

Sources, licenses and attribution

—Qwen/Qwen2.5-0.5B-Instruct at revision 7ae557604adf67be50417f59c2c2f167def9a775 (Apache-2.0).
—HuggingFaceTB/SmolLM2-360M-Instruct at revision a10cc1512eabd3dde888204e902eca88bddb4951 (Apache-2.0).
—Wikitext-2 raw test set, Merity et al., CC BY-SA 3.0; test file SHA-256 173c87a53759e0201f33e0ccf978e510c2042d7f2cb78229d9a50d79b9e7dd08, obtained from the canonical llama.cpp CI mirror (ggml-org/ci, revision 927b3642933080f1b0e811e2f916e14c292992f9).
—HellaSwag validation text, Zellers et al., MIT (dataset); file SHA-256 d572539320eb2050e858ca34b495bbe2103e3b3f1391a9c3bcdf215b0bb93bd1, from the klosax/hellaswag_text_data mirror at commit 5eba56b9ced146c37dc683bead159c5ebcd82cde.
—llama.cpp (MIT); container image ghcr.io/ggml-org/llama.cpp:full@sha256:5faf86f95747fbb40014a8b28c505d8ec0da3d983d4d368a6650bc056f252688, build 10991, commit 930e2fa59.
—Harness code in this directory is Apache-2.0 (see LICENSE).

Redistributed here: harness code, metrics, logs and hashes. Not redistributed: model weights and dataset text.

Agent use and accountability

This artifact was produced by AI agents of CyberNative AI LLC: agents designed the protocol, executed both runs and analyzed the results. The human founder owns the company and accountability for this publication; no human authored the runs.