NagaYu/echocache-guard-benchmark

EchoCache Guard Benchmark

131 labelled prompt pairs that measure the one thing a semantic cache must never get wrong: serving an answer written for a different question.

Every negative pair is a high-similarity negative — two prompts that differ in exactly one meaning-carrying element. Those are the pairs a similarity score cannot decide, and they are where a semantic cache turns into a wrong-answer generator.

Use it to score any cache or matcher, not only EchoCache.

Ground truth

expect_reuse = true   <=>  an answer written for `candidate` is a CORRECT answer to `query`
expect_reuse = false  <=>  serving the cached answer would be wrong or misleading

Fields

Relations

Positives — a correct cache should reuse these

Negatives — reuse would be a wrong answer

signoff_word_as_content exists because of a real defect found during development: boilerplate stripping anchored on a bare word turned Translate cheers into French into translate, colliding with every other Translate … prompt and serving a wrong answer through the exact-match path, where no guard runs. Any cache that strips sign-offs needs these five pairs.

Scoring

pip install numpy huggingface_hub
python3 evaluate.py                          # scores the EchoCache matcher from the Hub
python3 evaluate.py --matcher ./my_matcher.py

evaluate.py expects a module exposing match(query, candidate, threshold, use_guard) -> {"reuse": bool, "similarity": float}.

The headline metric is wrong reuse: the share of "must not reuse" pairs served from cache anyway. Reuse recall is only worth reading once wrong reuse is near zero — a matcher that reuses nothing scores a perfect 0% wrong reuse and is useless.

Baseline: EchoCache matcher v1.0.0

`NagaYu/echocache-matcher`, character 3-gram cosine + MismatchGuard:

The "guard off" rows are what a pure similarity threshold does on its own: at 0.80 it reuses everything it should — and 29 pairs it should not.

Remaining errors at 0.92, guard on (all three are the same shape, and they are misses, not wrong answers):

At 0.80 those three are reused correctly, and the single remaining error becomes signoff_word_as_content-004 (kind regards vs warm regards, similarity 0.891) — a wrong reuse the guard does not catch because the differing words are not proper nouns.

Provenance and honest caveats

• —Hand-authored synthetic pairs. These are not production traffic and contain no user data, no PII and no scraped content. They were written to isolate one variable at a time.
• —Not an independent evaluation. The benchmark was written by the same author, alongside the matcher it scores. Two matcher defects were found with this set and fixed in response to it (a boilerplate over-strip and a missing exclude negation cue), so the numbers above include that feedback loop. Treat it as a regression baseline and a diagnostic, not as a third-party leaderboard.
• —Small and deliberately adversarial. 131 pairs, weighted 2:1 toward negatives. Absolute accuracy figures do not transfer to your traffic; the wrong-reuse comparison between configurations is the part that generalizes.
• —Two languages. English and Japanese only. A cache for other languages needs its own pairs.
• —No cross-lingual pairs beyond unrelated — surface matchers score those at 0.00 and the case is uninteresting.

Extending it

Add rows to pairs.jsonl in the same shape. The most valuable additions are negatives from your own logs: two prompts your cache treated as the same and should not have. One real wrong hit is worth ten synthetic pairs.

The three repositories

• —Space — `NagaYu/EchoCache`: the running cache, dashboard and threshold sweep.
• —Model — `NagaYu/echocache-matcher`: the matching core as one numpy-only file.
• —Dataset — NagaYu/echocache-guard-benchmark: this benchmark.

Apache-2.0.