Hstre/evidence-k-registry

Evidence-k Registry

An open, versioned registry of measured evidence-saturation points for language-model deployments.

A model's context-window capacity is not the same as its optimal evidence budget. The registry records k\*: the empirically optimal number of decision-relevant evidence fragments for a stated combination of model, served backend, task type, context format, output contract, and reliability axis.

This is not a leaderboard. A higher k\* is not automatically better. The purpose is calibration: supplying the amount of evidence that maximizes the stated reliability objective instead of filling the available context window by default.

What one row means

Every row should be read as a property of a measured deployment:

(model, served backend, task type, context format, output contract, metric axis) -> k*

It is not a universal property of a model name. Quantization, provider routing, prompt structure, chunk size, task definition, and the selected reliability axis can all move the result.

Initial pilot data

The registry is seeded with the 35 task-level measurements reported in Table 4 of:

Hanns-Steffen Rentschler, _The Evidence-Saturation Point of Large Language Models_ (pilot working paper, v0.2, July 2026).

• —7 valid model/backend combinations
• —5 task types
• —k ladder: {0, 1, 2, 3, 5, 8, 13, full}
• —6 synthetic cases per task type
• —provider-pinned runs
• —correctness-optimal k\* values

The pilot found that the correctness-optimal k varied from 1 to full across task types for a fixed model. It also showed why correctness alone is insufficient: contamination or drift may bend underneath a flat correctness curve.

Paper: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=7038059 Reference implementation: https://github.com/hstre/evidence-k

Data files

• —data/registry.csv — accepted measurements displayed in the Dataset Viewer
• —data/exclusions.csv — attempted runs excluded from the registry and the reason for exclusion
• —templates/submission.csv — blank contribution template
• —SCHEMA.md — field definitions
• —CONTRIBUTING.md — submission and review rules

Contribution standard

Contributions are welcome through pull requests. A submission should:

1. 1.identify the exact model and served backend;
2. 2.report revision and quantization when known;
3. 3.define the task type, context format, output contract, k ladder, and metric axis;
4. 4.report case count and repetitions;
5. 5.provide code, configuration, run logs, or another inspectable source;
6. 6.distinguish measured values from estimates;
7. 7.state exclusions and failed runs rather than silently removing them.

Results without enough provenance to interpret the number will not be merged into the main table.

Limitations

The seed measurements are a pilot, not definitive performance estimates. They use small synthetic datasets, a limited k ladder, and a heuristic contamination instrument. No confidence intervals are claimed. The registry is intended to make broader replication possible, not to turn the pilot into a universal law.

Citation

@article{rentschler2026evidence,
  title   = {The Evidence-Saturation Point of Large Language Models},
  author  = {Rentschler, Hanns-Steffen},
  year    = {2026},
  month   = {July},
  note    = {Pilot working paper, version 0.2},
  url     = {https://papers.ssrn.com/sol3/papers.cfm?abstract_id=7038059}
}