pbhappliedsystems/quant_eval_efficiency_and_footprint

quant_eval — Efficiency and footprint

One row per published run: stored weight artifact bytes before and after quantization, compression ratio, observed evaluation wall-time ratio with an explicit direction label, the accelerator used on each lane, and token throughput.

Part of the quant_eval public corpus: a per-case behavioral evaluation of full-weight and quantized large language models across eight agent-relevant task families, with paired statistical testing.

Cite this dataset: 10.5281/zenodo.22010723 — concept DOI, always resolves to the latest version. This exact deposit: 10.5281/zenodo.22806121 — version DOI, frozen. Cite this one where reported numbers must stay verifiable against the object referenced.

Version 1.0.1 corrects a material omission in 1.0.0. See What changed in 1.0.1 below before using any runtime figure from the previous version.

What this file contains

Supporting files: source_bundle_checksums.json.

Corpus scope

Every run evaluates a full-weight baseline and a quantized variant of the same model against the identical locked fixture set, case for case. Statistical comparison is paired: the two-sided exact McNemar test on per-case outcomes, with Wilson intervals on the rates.

What changed in 1.0.1

Version 1.0.0 recorded the execution substrate — local_llama_cpp or modal_llama_cpp — but did not record which accelerator ran each lane of a pair. On the two Modal runs the full-weight lane and the quantized lane executed on different accelerator classes. Their wall-time ratios of 0.961 and 0.854 therefore measure a hardware change at least as much as a precision change, and 1.0.0 presented them as if quantization alone produced them.

Three consequences, all corrected here:

1. 1.Three columns are added — baseline_accelerator, quantized_accelerator, hardware_matched_pair — so the asymmetry is visible in the data rather than only in prose.
2. 2.The wall-time figure is regenerated with accelerator labels on every bar and the unmatched pairs marked as such.
3. 3.The interpretive sentence in 1.0.0 is withdrawn. It read that quantization "buys latency only on some substrates." The corpus does not support that. What it supports is stated under Runtime below.

No source bundle changed. No measured value changed. source_bundle_checksums.json is byte-identical to 1.0.0, and all 72 source digests still verify.

Runtime — read this before using any ratio

Observed, on matched hardware. Four pairs ran both lanes on the same accelerator. On those, the quantized variant completed the evaluation faster than full weight in every case, at ratios of 2.704, 2.564, 1.985 and 1.714. This is the expected direction: a smaller weight artifact moves less data per token, and decode is memory-bandwidth bound.

Observed, on unmatched hardware. Two pairs ran the full-weight lane and the quantized lane on different accelerator classes. Their ratios, 0.961 and 0.854, are not attributable to precision. They are reported because they were measured, and they are labelled hardware_matched_pair = false so they can be filtered out of any precision comparison.

Reasoned, not measured here. Held on a single accelerator, a quantized variant is expected to complete faster than its full-weight baseline, for the memory-bandwidth reason above. The four matched pairs in this corpus are consistent with that expectation. This dataset does not contain a controlled same-hardware comparison for the two Modal models, so that expectation is stated as a reasoned one and not as a finding of this corpus.

Run-to-run variance is substantial and is not modelled. The identical 24,504,279,808-byte Mistral-Nemo F16 artifact recorded 25.04, 30.14 and 33.11 tokens per second across its three runs — a 32.2% spread on the same file — while producing behaviourally identical output, 0 of 1,600 cases differing. Wall-time ratios in this dataset are therefore single observations carrying at least that much variation, and small differences between them should not be read as real.

Figures

[image]

Observed evaluation wall-time ratio (full weight divided by quantized) for all six published pairs, with the accelerator shown for each lane. The two bars below parity ran their lanes on different accelerator classes; their position reflects that hardware difference and is not a measured effect of quantization. Observed harness wall time on the recorded hardware and backends; not a controlled throughput benchmark and not a general claim about quantization performance at any precision on any hardware.

Figures are generated directly from the harness rollups by the published build tooling; no plotted value is recomputed, smoothed, or fitted.

Columns

quant_eval_efficiency_and_footprint.csv

  run_id                                        model_id                                      baseline_kind
  quant_type                                    baseline_bytes                                quantized_bytes
  compression_ratio                             size_reduction_fraction                       measurement_scope
  excludes                                      baseline_runner                               quantized_runner
  execution_substrate                           baseline_accelerator                          quantized_accelerator
  hardware_matched_pair                         baseline_tokens_per_second                    quantized_tokens_per_second
  observed_wall_time_ratio                      wall_time_direction                           throughput_ratio
  token_volume_cost_proxy_ratio                 monetary_cost_ratio_status                    artifact_footprint_status
  hardware_backend_scope

hardware_matched_pair is the field to filter on. true means both lanes of that pair ran on the same accelerator class and the ratio is a precision comparison. false means they did not and it is not.

Verification

This corpus is derived from sanitized publication bundles produced by the quant_eval harness. It is designed to be checked rather than trusted:

• —source_bundle_checksums.json, included here, republishes, verbatim, the SHA-256 digest and byte length of every file in every source bundle. No source file was modified.
• —Before this file was written, the builder verified all 72 source-file digests and independently recomputed all 96 family x runner pass rates from the raw per-case rows, matching the harness rollups exactly.
• —The figures here are not derived from the per-case results dataset. Stored-artifact byte counts and observed wall time are recorded by the harness at run time and are carried through from the source bundles unchanged; they are traceable through the bundle digests above, not recomputable from per-case rows. Pass-rate aggregates, which are recomputable, live in the paired degradation statistics (D5) and family pass rates (D6) datasets.
• —The accelerator fields added in 1.0.1 are declared, not harness-recorded. The source bundles do not capture accelerator class, so these values come from a declaration file published with the build tooling. They are traceable to that declaration rather than to a bundle digest, and that distinction is deliberate.

Limits you should know before using this

• —Two of the six pairs are hardware-unmatched. Their full-weight and quantized lanes ran on different accelerator classes, so their wall-time ratios are not precision comparisons. Filter on hardware_matched_pair before comparing anything. This is the single most important limit in this dataset.
• —Runtime figures are observed harness wall time on the recorded hardware and backends, carrying at least the 32.2% run-to-run variance documented above. They are not a controlled throughput benchmark and not a general claim about quantization performance at any precision on any hardware. Direction is published as an explicit label because not every measured pair is a speedup.
• —Compression is measured on the stored weight artifact only. excludes records modal_container_storage;runtime_vram on every row: runtime VRAM is not measured anywhere in this dataset. Do not read compression ratio as a VRAM figure.
• —The monetary cost ratio is suppressed in every row because per-runner cost was not recorded. Do not read the token volume proxy as a price.
• —Decoding conditions are not uniform across models. Temperature follows each publisher's own model card, so cross-model comparison of absolute pass rates is confounded. Within-run pairing is unaffected, which is what the paired test requires. The conditions are published per row and per run so they can be filtered on.
• —Runs on the Modal substrate record `seed` status `unsupported`, because the deployed method signature accepts no seed parameter. This records a configuration fact, not a determinism fact: repeat runs of the same model, version and fixture set produced identical output. Local runs applied a fixed seed explicitly.
• —The fuzz family is an adaptive trajectory evaluated from identical starting fixtures. Its paired test compares complete case outcomes, not identical post-divergence prompts.
• —Calibration runs are not published. Runs that informed a published run are disclosed by identifier in calibration_lineage.csv, published in the run provenance dataset (D4), so the record is complete without releasing provisional numbers.

Citation

@dataset{pbh_quant_eval_d7,
  author    = {Hill, Patrick},
  title     = {quant_eval Efficiency and footprint},
  publisher = {PBH Applied Systems, LLC},
  year      = {2026},
  version   = {1.0.1},
  doi       = {10.5281/zenodo.22010723},
  note      = {Version DOI: 10.5281/zenodo.22806121},
  license   = {CC-BY-4.0}
}

Licence

Creative Commons Attribution 4.0 International (CC BY 4.0). See LICENSE. Commercial use is permitted; attribution is required.

This corpus describes third-party models and redistributes no model weights. Each evaluated model remains under its own licence, recorded per run in the run provenance dataset.

Produced by builddatasets.py 2.5.0 from quanteval publication bundles. Built 2026-09-17.