tedbelford/cliffs-and-slopes-semantic-cache

Cliffs and Slopes — Semantic-Cache Benchmark

Benchmark for the paper "Cliffs and Slopes: Why Semantic Caches Fail Structurally, Not Just Through Miscalibration" (under submission to Information and Software Technology).

A semantic cache answers a new query with the stored answer of a "similar enough" past query. This benchmark measures where — and why — threshold-on-similarity caching fails structurally: the task requires an equivalence relation over queries, but is implemented as a threshold on smooth, non-transitive similarity.

What's in the dataset

• —46,214 query pairs (seed question, variant) after dual-judge filtering and MinHash deduplication — 51.8% HIT.
• —5 domains: university admin (EN), academic NLP/Qasper (EN), tech docs (EN), medical (EN), Vietnamese administrative/education (VI).
• —2 languages: English and Vietnamese.
• —9 transformation axes in 3 groups, derived from a single organizing principle — the ratio answer-step / surface-step:

• —Labels are construction-derived, never model-assigned: the axis determines expected_label (P→HIT, N→MISS) by code. Two LLM judges (primary judge ≠ generator model, always) assess axis compliance only.
• —Independently audited on a stratified 1,500-pair subset: three blind human annotators, Fleiss' κ = 0.76, majority label vs construction label 98.5% (see Human audit below; raw annotations included).

Files

Record fields

{
  "id": "d1_s0001_N1_v1",
  "domain": "university_admin",
  "seed_id": "d1_s0001",
  "seed_question": "...",     // the cached query
  "answer": "...",            // answer to the seed question
  "variation": "...",         // the incoming query
  "axis": "N1", "axis_group": "slope",
  "expected_label": "MISS",   // derived from axis by code
  "delta": "...",             // what changed
  "generator_model": "claude-opus-5",
  "judge1": {"model": "claude-sonnet-5", "axis_ok": true, "conf": 0.95, "issue": null},
  "judge2": {"model": "claude-opus-5",  "axis_ok": true, "conf": 0.92, "issue": null},
  "batch_id": "A4_0009", "ts": "2026-08-12T15:00:59Z"
}

Dataset analysis

Composition — pairs per axis × domain (46,214 total, 51.8% HIT):

[image]

The designed spectrum, recovered from data — median surface step (1 − word Jaccard) vs. embedding distance (1 − cosine, BGE-M3) per axis. Embeddings track surface form: the cliff axes (orange — where the answer flips) sit closest to the origin, i.e. closest to the same-answer region:

[image]

Where the irreducible overlap lives — cosine-similarity distributions of each different-answer axis (colored) against the same-answer distribution (green), BGE-M3. Cliff-axis pairs sit deep inside the same-answer distribution — no threshold separates them:

[image]

These figures are produced by scripts/make_dataset_figs.py in the accompanying code repository, directly from data/benchmark.jsonl.

How it was built

Seeds were extracted from licensed source documents (claude-sonnet-5), variants generated per axis (claude-opus-5; RQ6 stability subsets regenerated with sonnet and haiku), then filtered by two independent axis-compliance judges and MinHash-deduplicated. Complete generation logs (full prompts, raw outputs, exact model strings, timestamps, prompt-version hashes) are preserved in the accompanying code repository. Raw source documents are not redistributed; per-document provenance (URL, license, retrieval date) ships with the code repository.

Human audit of the labels

A 1,500-pair subset stratified over the 9 axes × 5 domains grid (164–168 per axis, ~300 per domain) was labeled by three blind human annotators working independently (university-educated volunteers; all read Vietnamese, as ~300 pairs require). Each annotator labeled all 1,500 pairs, seeing only the two questions (A/B order randomized) — never the construction label, the axis, the judge outputs, a source passage, or a reference answer — and answered one question per pair: would one complete, correct answer serve both queries, assuming a shared knowledge source? Because no source material was shown, the audit certifies question-level answer substitutability, not extensional answer coincidence under the corpus (see the paper's Threats section); a source-grounded round-2 kit ships in the code repository (human_annotation_v2/).

Analysis rules: three pairs (all N5) were rated UNSURE by two annotators — no 2-of-3 definite majority — and are excluded from all statistics, so agreement and accuracy use the same n = 1,497. Pairs cluster by seed (1,060 seeds), so headline CIs are percentile cluster bootstraps over seeds (10,000 replicates).

The 22 disagreements concentrate on N2 and N5 (6 each) and run 17:5 toward humans judging an expected-MISS pair same-answer — an observed human–construction disagreement rate of 1.5%, far below the measured irreducible overlap (0.19–0.50). Reproduce every number with scripts/gold_agreement.py in the code repository (output mirrored here as human_annotation/human_gold.json).

Intended use

Evaluating embedding models and cache architectures on answer-equivalence decisions, studying order-dependence/non-transitivity, and threshold-calibration research. Not a QA-accuracy benchmark: answers are grounded in specific source documents and may age.

Licensing (read this)

Mixed, following the source documents (full table in DATASHEET.md):

• —`domain == "university_admin"` (d1, derived from MIT OpenCourseWare): CC BY-NC-SA 4.0 — ShareAlike/NonCommercial carries to derived pairs.
• —All other domains (d2–d5): CC BY 4.0.

Filter out university_admin if you need a purely CC BY subset.

Limitations

• —LLM-generated text with LLM compliance filtering; labels independently audited on a stratified 1,500-pair subset (κ = 0.76, 98.5% agreement — see above), while the remaining pairs carry construction-derived labels only. The audit is question-pair-only (no source grounding); a source-grounded round-2 kit is prepared. Larger-scale annotation remains invited.
• —The three annotators are university-educated colleagues of the authors (unpaid, blind to labels, axes, and hypotheses) — not an external crowd.
• —N4/N5 applicability is seed-conditional (overgenerated ×4, top-2 judge-passed kept).
• —Vietnamese domain drawn from the legal/administrative register.
• —Generator family is Claude-only.

Code & reproduction

Analysis code, cache simulation, experiment configs, generation logs, and scripts reproducing every figure and table: see the accompanying GitHub repository (semcache-bench).

Citation

Paper under submission; citation entry will be added upon acceptance. Until then, please cite this dataset by its Hugging Face URL.