pbhappliedsystems/quant_eval_v7_21_per_case_results_and_run_provenance

quant_eval v7.21 — Per-Case Evaluation Results and Run Provenance

Supplementary evidence for the whitepaper quant_eval: A Behavioral Evaluation Harness for Full-Weight and Quantized Large Language Models.

Author: Patrick Hill, PBH Applied Systems, LLC ORCID: 0009-0008-3662-1681 Licence: CC BY 4.0 Concept DOI (all versions): 10.5281/zenodo.22851375 Version DOI (this deposit): 10.5281/zenodo.22851376

What this deposit is

Every quantitative result reported in the whitepaper comes from two evaluation runs of quant_eval version 7.21. This deposit contains the complete per-case output of both runs and the provenance manifest each run wrote, so that any pass rate, bucket score, or latency figure in the paper can be recomputed from source and any individual failure inspected at the row level.

It contains 126 scored cases across three inference backends, 72 recorded signals per case.

What this deposit is not

This is not part of the quant_eval public corpus. The corpus (datasets D0–D7) reports production evaluations under quant_eval v7.22 and is deposited separately. Nothing here should be merged with it, compared against it, or read as superseding it.

The methodology used here was retired. Both runs used the fast_gate screening profile: a small per-family case count intended to triage models quickly, not to establish statistical confidence. The harness itself marks every family verdict in these runs provisional (n<10), and that qualifier is not decoration. fast_gate was retired immediately after the whitepaper was written and is not the methodology behind any published corpus dataset.

These are screening results, not production results. They are published because the whitepaper rests on them and a reader is entitled to check the arithmetic — not because they are a fit basis for a deployment decision about either model.

No model weights are redistributed here. The evaluated models remain under their own licences.

Files

Both manifests were written by the harness under the filename run_manifest.json. They are renamed here by run identifier because a flat archive cannot hold two files of the same name; the original filename is recorded inside each file.

The two runs

Both runs executed the identical fixture set:

golden_oracle_fixtures_v7_21__sha256_6d71a0b9147c.json
sha256 = 6d71a0b9147c079371b02a94f3c149eb78a6adc03dc16ff6833b964fbf4174f0

Each runner saw the same 42 cases across eight families: json (4), json_multistep (5), stateful_followup (2), mixed_brief_json (2), mcq (5), toolcall (2), toolcall_only (2), fuzz (20). The authored battery is 15 cases; mcq, toolcall and the 20-case fuzz property-based regression suite bring each runner's total to 42.

Decoding

Both manifests record "seed": 42. The decoding temperature is not recorded in these files. The harness applied sampled decoding at the temperature recommended on the Mistral-Nemo-Instruct-2407 model card, 0.3, together with that card's other recommended parameters.

The CSVs corroborate that sampling — not greedy decoding — was in use. All 15 json_multistep rows across both runs carry mode = best_of_k, best_of_k_k = 3, best_of_k_used = 1, with rotating per-case seeds (1337, 1338, 1437, 1537, 1637, 1638, 1737). In two of those rows the selector chose candidate index 1 rather than 0 — ms_easy_01 under full_weight_transformers (recorded similarity 0.682) and ms_med_02 under quantized_llama_cpp (0.000). Greedy decoding yields one deterministic completion per prompt, so three distinct candidates could not have existed to choose among.

Results

Pass rate is the fraction of rows in a family where every gating signal equals 1:

json, fuzz and mcq report bucket_score rather than a conjunctive pass rate; their pass column reads n/a.

Side-by-side run 20260211_022944

AWQ run 20260629_184555

Every figure above was recomputed directly from the deposited CSVs and matches the rollup the harness produced at run time.

Wall-clock timings, and why they are not a precision comparison

Mean per-case latency over all 42 cases, computed from the secs column:

These three numbers span three different inference backends. The spread between them combines a backend change with a precision change and cannot be attributed to quantization alone. They are evaluation-harness measurements under a fixed configuration, not throughput benchmarks, and they are published here so the whitepaper's figures can be traced to their source — not as a performance claim.

Recorded model artifacts

The side-by-side manifest records both GGUF artifacts with byte counts and SHA-256 digests:

That is a 3.277× reduction in stored artifact size — a property of the files themselves, independent of backend.

The AWQ manifest records no artifacts ("artifacts": []). No file digest exists for the AWQ model. Artifact-level verification is therefore possible for the FP16/Q4KM pair and not for the AWQ lane.

Cache reuse in the side-by-side run

The full-weight lane of run 20260211_022944 did not re-execute. Its results were reused from the cache written by an earlier run. The manifest records this:

"full_weight_transformers": {
  "cache_reuse_path": "<WORK_ROOT>/runs/.../Mistral_Nemo_Instruct_2407_20260211_014604/full_weight_cache.json",
  "cache_write_enabled": 0,
  "reused_cache": 1
}

The CSV corroborates it. The 42 full-weight rows carry only two distinct timestamp values, 20260211_023003 and 20260211_023004 — one second apart. Forty-two cases averaging 30.24 s cannot execute in two seconds; they were written out from cache in a single pass. The 42 quantized rows span 24 distinct timestamps across roughly ninety seconds of real execution.

The full-weight rows carry run_id = 20260211_022944 while the quantized rows carry run_id = Mistral_Nemo_Instruct_2407_20260211_022944. The two spellings are an artifact of the two lanes being written by different code paths and are left exactly as recorded.

The comparison remains sound — the fixture set, the seed and the prompts are identical across the source run and the side-by-side run — but the timings and scores in the full-weight lane were measured in run 20260211_014604, and a reader should know that.

Manifest sanitization

The deposited manifests are not byte-identical to the files the harness wrote. quanteval is proprietary and unpublished; its directory layout is not disclosed. Each machine-local filesystem root was replaced with a symbolic token — `<WORKROOT>, <CATALOGROOT>`, `<TOOLCHAINROOT>, <MODELLIBRARY>`, `<ALLOWEDROOT>` — and the path structure below each root was left verbatim, so cross-run references such as the cache source above remain traceable.

Fourteen strings were altered in the side-by-side manifest and ten in the AWQ manifest. Each file carries a _sanitization block recording the tool, the SHA-256 of the unmodified source file, and the count of altered strings.

No run identifier, timestamp, seed, fixture hash, artifact byte count, artifact digest, toolchain digest, runner role, quantization type, or cache-reuse field was altered. The transformation was applied programmatically under two abort conditions: any surviving machine-local path, or any change to a field on the preserved list.

The CSVs are deposited exactly as the harness wrote them. Nothing in them was modified, reordered, or removed.

CSV schema

Both files share an identical 72-column header. Orientation for the columns that carry most of the weight:

bucket_score scales are family-specific and are not comparable across families. For json_multistep a 10 requires plan_exact_match, checks_ok and final_match_reported together; otherwise the score is the number of leading plan steps matching the oracle. mcq runs 0–1. The rest run 0–10.

For json_multistep, Tier 1 (gating) covers planning feasibility and externally verified outcome correctness; Tier 2 (final_consistent_ok, final_match_reported) tracks final-state reporting discipline and does not gate a Tier 1 pass. Both Tier 2 rates are 0.000 across all three runners in these runs.

How to verify

import csv, statistics
csv.field_size_limit(2**31 - 1)

rows = list(csv.DictReader(
    open("comparison_results_v7_21_Mistral_Nemo_Instruct_2407_20260211_022944.csv",
         newline="", encoding="utf-8")))

gating = ["schema_ok", "checks_consistent_ok", "stop_semantics_ok", "oracle_equiv_ok"]

for runner in ("full_weight_transformers", "quantized_llama_cpp"):
    sel = [r for r in rows if r["runner"] == runner and r["family"] == "json_multistep"]
    passed = sum(1 for r in sel if all(r[c] == "1" for c in gating))
    print(runner, passed / len(sel))            # 0.6 then 0.4

for runner in ("full_weight_transformers", "quantized_llama_cpp"):
    sel = [r for r in rows if r["runner"] == runner]
    print(runner, round(statistics.mean(float(r["secs"]) for r in sel), 3))   # 30.244 / 1.417

Model artifacts can be checked against the digests in run_manifest_20260211_022944.json, where the source files are available.

Limitations

1. 1.Sample sizes are small by design. Four of eight families have N ≤ 5 per runner. The harness labels every verdict in these runs provisional; treat them as screening signal.
2. 2.Backends differ across the three runners. No same-backend precision comparison exists in this data.
3. 3.One worked model. Everything here concerns Mistral-Nemo-Instruct-2407. Nothing generalizes to other model families without re-running.
4. 4.The full-weight lane was cache-reused, as recorded above.
5. 5.No artifact digest exists for the AWQ model.
6. 6.Fixture contents are not included. The CSVs record each case's expected value and the model's output, not the prompts. The fixture file is identified by SHA-256 only.
7. 7.The scoring code is not published. quant_eval is proprietary. These files support recomputation of the reported statistics from recorded signals; they do not permit re-deriving the signals from raw outputs.

Citation

Hill, Patrick. quant_eval v7.21 — Per-Case Evaluation Results and Run Provenance. PBH Applied Systems, LLC, 2026. CC BY 4.0. DOI: 10.5281/zenodo.22851375

See CITATION.cff for machine-readable metadata.

Related

• —Whitepaper — quant_eval: A Behavioral Evaluation Harness for Full-Weight and Quantized Large Language Models. This deposit is its evidence base. Concept DOI 10.5281/zenodo.22851592.
• —quant_eval public corpus (D0–D7) — production evaluations under v7.22, deposited separately. Concept DOI 10.5281/zenodo.22009419. Different harness version, different methodology, different corpus. Do not conflate the two.

PBH Applied Systems, LLC — Oklahoma City, Oklahoma.