notmehul/memory-bench

memory-bench v1.0.0, public release

A screened benchmark for organizational memory in agent harnesses: does a memory system keep a rule that was stated once, drop a fact that was superseded, and pick the right one when tiers conflict?

371 valid paired probe instances across 3 simulated organizations, drawn from 486 probes over 612 events. Scored as pair credit: an instance counts only if the base task and its counterfactual twin both pass, so anything answerable from priors earns nothing.

Layout

data/org-0000N/        events.jsonl    the stream, one event per line
                       probes.jsonl    work tasks + scoring assertions
                       g3-report.json  the frozen valid set
                       org.json        personas/teams/event witness index
                                       (ground-truth ledger REDACTED)
data/org-0000N-twin/   events.jsonl    counterfactual stream
                       org.json        redacted, as above
croissant.json         dataset metadata (Croissant 1.0)
rai.json               responsible-AI metadata
CHECKSUMS.txt          sha256 of every shipped file
MANIFEST.json          what shipped, what was withheld, and why
MAINTENANCE.md         versioning + the re-anchoring procedure
LICENSE-DATA           CC BY 4.0

What is withheld, and why

The ground-truth fact ledger (746 facts across the public seeds) stays private: it records which facts are probed and which are planted distractors, so publishing it would let a system learn the generator's planting patterns rather than remember the organization. The generator and the two unscreened holdout seeds are withheld for the same reason. They are the answer if the public seeds are ever contaminated.

The probe assertions ARE published. Scoring is impossible without them, and an open benchmark that hides its criteria cannot be checked. This makes the set open-book by construction: a system tuned against these criteria will score well here and generalize nowhere, and the canary strings embedded in every stream are how you detect one that trained on this data.

A warning about the anchors

Every instance's validity was established under one pinned worker model, which its provider has since deprecated. Probe validity is task-model-relative: we measured 37/54 versus 17/54 cluster survival across two workers under identical rules. The valid set shipped here is valid relative to that worker, not in the abstract. Running a different worker without re-screening produces numbers that do not mean what they appear to mean. MAINTENANCE.md gives the procedure.

Papers and code

• —Dataset and validity study: doi:10.5281/zenodo.22838320
• —Construction methodology: doi:10.5281/zenodo.22838602
• —Harness, screening and scoring, and the evidence behind every number: github.com/notmehul/memory-bench
• —Development record: github.com/notmehul/memory-bench-history

Citation

Srivastava, M. (2026). memory-bench: A Screened Benchmark Dataset and Validity Study for Organizational Memory in Agent Harnesses. Zenodo. https://doi.org/10.5281/zenodo.22838320

License

Data: CC BY 4.0 (LICENSE-DATA). Code in the repository: MIT.