pbhappliedsystems/quant_eval_throughput_telemetry

1---2license: cc-by-4.03language:4- en5pretty_name: >-6  quant_eval Throughput telemetry7size_categories:8- 10K<n<100K9tags:10- quantization11- large-language-models12- gguf13- agent-evaluation14- tool-calling15- behavioral-evaluation16- mcnemar17- llama-cpp18- model-evaluation19- reproducibility20annotations_creators:21- machine-generated22source_datasets:23- original24configs:25- config_name: default26  data_files:27  - split: train28    path: quant_eval_throughput_telemetry.jsonl29---30 31# quant_eval — Throughput telemetry32 33**One record per generation call across all published runs, pooled into a single union schema. Carries token counts, timing, and the token-count source per call.**34 35Part of the quant_eval public corpus: a per-case behavioral evaluation of full-weight and quantized large language models across eight agent-relevant task families, with paired statistical testing.36 37**Cite this dataset:** [10.5281/zenodo.22009987](https://doi.org/10.5281/zenodo.22009987) — concept DOI, always resolves to the latest version.38**This exact deposit:** [10.5281/zenodo.22009988](https://doi.org/10.5281/zenodo.22009988) — version DOI, frozen. Cite this one where reported numbers must stay verifiable against the object referenced.39 40## What this file contains41 42| File | Rows | Columns |43|---|---:|---:|44| `quant_eval_throughput_telemetry.jsonl` | 27,370 | 31 |45 46Supporting files: `data_dictionary.json`, `source_bundle_checksums.json`.47 48## Corpus scope49 50| Run | Model | Baseline | Quantized | Substrate | Licence |51|---|---|---|---|---|---|52| `Mistral_Nemo_Instruct_2407_20260814_030505` | mistralai/Mistral-Nemo-Instruct-2407 | gguf_f16 | gguf_q4_k_m | local | Apache-2.0 |53| `Mistral_Nemo_Instruct_2407_20260815_113254` | mistralai/Mistral-Nemo-Instruct-2407 | gguf_f16 | gguf_q5_k_m | local | Apache-2.0 |54| `Mistral_Nemo_Instruct_2407_20260816_084553` | mistralai/Mistral-Nemo-Instruct-2407 | gguf_f16 | gguf_q8_0 | local | Apache-2.0 |55| `Qwen2.5_14B_Instruct_1M_20260815_220633` | Qwen/Qwen2.5-14B-Instruct-1M | modal_f16 | modal_q4_k_m | Modal | Apache-2.0 |56| `Qwen2.5_32B_Instruct_20260815_081051` | Qwen/Qwen2.5-32B-Instruct | modal_f16 | modal_q4_k_m | Modal | Apache-2.0 |57| `Qwen2.5_7B_Instruct_20260814_234822` | Qwen/Qwen2.5-7B-Instruct | gguf_f16 | gguf_q4_k_m | local | Apache-2.0 |58 59Every run evaluates a full-weight baseline and a quantized variant of the same model against the identical locked fixture set, case for case. Statistical comparison is paired: the two-sided exact McNemar test on per-case outcomes, with Wilson intervals on the rates.60 61## Columns62 63### `quant_eval_throughput_telemetry.jsonl`64 65```66  attempt                                       call                                          call_id67  case_id                                       cumulative_generated_tokens                   cumulative_generation_seconds68  cumulative_prompt_tokens                      cumulative_tokens_per_second                  evaluation_contract_id69  event                                         family                                        finish_reason70  fixtures_sha256                               generated_tokens                              generation_seconds71  max_new_tokens                                max_new_tokens_recorded                       model_id72  prompt_tokens                                 quant_type                                    run_elapsed_seconds73  run_id                                        runner                                        schema_version74  scored                                        stage                                         timestamp_unix75  token_count_source                            token_limit_reached                           tokens_per_second76  version_tag77```78 79`data_dictionary.json`, included here, documents every field: applicability, data type, evidence role, gate membership, the meaning of an empty value, and observed population counts. **An empty cell is not automatically a missing measurement** — several fields are conditional on task family, and their empty-value meaning is recorded explicitly.80 81## Verification82 83This corpus is derived from sanitized publication bundles produced by the quant_eval harness. It is designed to be checked rather than trusted:84 85- `source_bundle_checksums.json`, included here, republishes, verbatim, the SHA-256 digest and byte length of every file in every source bundle. No source file was modified.86- Before this file was written, the builder verified all 72 source-file digests and independently recomputed all 96 family x runner pass rates from the raw per-case rows, matching the harness rollups exactly.87- Aggregates published elsewhere in this corpus — the paired degradation statistics (D5) and family pass rates (D6) datasets — are recomputable from the per-case results dataset (D1) using the gate definitions in `data_dictionary.json`, included here.88 89## Limits you should know before using this90 91- **Decoding conditions are not uniform across models.** Temperature follows each publisher's own model card, so cross-model comparison of absolute pass rates is confounded. Within-run pairing is unaffected, which is what the paired test requires. The conditions are published per row and per run so they can be filtered on.92- **Runs on the Modal substrate record `seed` status `unsupported`** — the deployed method signature accepts no seed — so those runs are not exactly reproducible. Local runs applied a fixed seed.93- **Runtime figures are observed harness wall time** on the recorded hardware and backends. They are not a controlled throughput benchmark and not a general claim about quantization performance at any precision on any hardware. Direction is published as an explicit label because not every measured pair is a speedup.94- **The fuzz family is an adaptive trajectory** evaluated from identical starting fixtures. Its paired test compares complete case outcomes, not identical post-divergence prompts.95- **Calibration runs are not published.** Runs that informed a published run are disclosed by identifier in `calibration_lineage.csv`, published in the run provenance dataset (D4), so the record is complete without releasing provisional numbers.96 97## Citation98 99```bibtex100@dataset{pbh_quant_eval_d2,101  author    = {Hill, Patrick},102  title     = {quant_eval Throughput telemetry},103  publisher = {PBH Applied Systems, LLC},104  year      = {2026},105  doi       = {10.5281/zenodo.22009987},106  note      = {Version DOI: 10.5281/zenodo.22009988},107  license   = {CC-BY-4.0}108}109```110 111## Licence112 113Creative Commons Attribution 4.0 International (CC BY 4.0). See `LICENSE`. Commercial use is permitted; attribution is required.114 115This corpus describes third-party models and redistributes no model weights. Each evaluated model remains under its own licence, recorded per run in the run provenance dataset.116 117---118 119Produced by build_datasets.py 2.4.0 from quant_eval publication bundles. Built 2026-08-19.120