arjhinety/small-mind-pmb-v0

PMB v0 — Personalised Memory Benchmark

An evaluation benchmark for long-horizon personalised memory in small language models. It asks whether a model can recall what a specific user told it across many sessions, and — the part most memory benchmarks skip — whether it can decline to answer when the memory does not contain the answer.

Built for small-mind-companion, a study of how much of the long-horizon memory gap a ~2B multimodal model can close without scaling parameters.

Part of the OneBee Datasets collection.

Composition

Per category: factual 98, continuity 96, distractor 96, outdated_fact 96, preference 91, episodic 69, temporal 62, unanswerable 80.

Each probe carries probe_id, persona_id, question, category, answerable, gold_answer, gold_supporting_memory_ids, and acceptable_alternatives. All 80 unanswerable probes correctly carry no gold answer.

Files

Directory names and contents are copied verbatim from the source repository, so the integrity pins below are reproducible with that repository's own hashing script rather than with a re-implementation.

pmb_v0_full/                   the benchmark proper
  probes.jsonl                 688 probes
  personas/p000.json …         8 persona definitions (the user's synthetic identity)
  DATASHEET.md                 generator record
  hash.txt                     90026a53… — covers every file in this directory

pmb_v0/                        a 2-persona, 36-probe smoke corpus from the same generator
  probes.jsonl                 36 probes
  personas/p000.json, p001.json
  DATASHEET.md
  hash.txt                     9089c7bb… — covers every file in this directory

memory_stores/pmb_v0_full/     8 SQLite memory stores the probes are answered against
  p000.db … p007.db            no pin exists for these in the source repository either

The memory stores are required for reproduction. gold_supporting_memory_ids point into them, so without the stores you can read the questions but cannot run the system under test. They are SQLite databases with an FTS5 index — open with any SQLite client.

Verifying the pins

Both hash.txt files are benchmark_hash values from the source repository: SHA-256 over every file in the directory, visited in sorted order, feeding the LF-normalised relative POSIX path followed by the LF-normalised file bytes, and skipping hash.txt itself. From a clone of small-mind-companion at tag study-001:

uv run python scripts/recompute_hashes.py        # re-derives and compares all 11 corpora

pmb_v0_full here corresponds to data/benchmarks/pmb_v0_full, pmb_v0 to data/benchmarks/pmb_v0, and memory_stores/pmb_v0_full to data/stores/pmb_v0_full.

Intended use

Evaluating a retrieval-augmented or memory-augmented system on personalised recall and abstention. Score per category rather than in aggregate: the aggregate hides the tradeoff the benchmark exists to expose.

Limitations — read these

• —Generated by a live model, not human-written, and not human-reviewed. The probes were produced by an OpenAI-compatible teacher endpoint. No reviewer log was retained, so the "not human-reviewed" status is self-reported and cannot be verified from any artifact.
• —`acceptable_alternatives` is present on every probe and empty in 688/688. Exact-match scoring (pra_strict) is therefore ~0 by construction, and any system will look like it scores zero on it. The released metric is judge-scored pra_lenient (an LLM judge, ≥3/5), reported in the source project's docs. If you intend to use pra_strict, populate this field first.
• —8 personas is small. The 64/64 cell design is balanced, but eight identities cannot support claims about population-level behaviour. Treat per-persona results as descriptive.
• —One known contamination question is answered, not assumed. The source project's SFT corpora were checked against these probes with a 13-gram overlap test and are clean; the check is reproducible with scripts/check_contamination.py.
• —*The training corpora share persona names with this set* by accident of a shared name pool. IDs, facts and conversations differ. Do not use a training-persona corpus as a held-out set.
• —The memory stores carry no integrity pin, because data/stores/ has no hash.txt in the source repository. They ship because reproduction needs them, not because they are pinned.

Citation

@misc{small_mind_pmb_v0,
  title  = {PMB v0: a personalised-memory benchmark with abstention probes},
  author = {Ty, Arjhine A.},
  year   = {2026},
  note   = {Part of the small-mind-companion Study 001 release},
  url    = {https://huggingface.co/datasets/arjhinety/small-mind-pmb-v0}
}

Provenance

Read from `small-mind-companion` at the frozen tag `study-001` (commit bc80d59, frozen 2026-09-13). Study 001 is frozen: its evidence is hash-pinned in `reports/data/study-001-freeze.json` and its write-up is at small-mind.arjhinety.com.

This repository publishes its own corrections. Thirty-four claims the committed artifacts did not support are documented in `reports/ERRATA.md`, including one that affected this data. Read that before relying on anything here.

Licence

Data and benchmark content: CC-BY-4.0. Code in the source repository: Apache-2.0.