ris3abh-11/engram-eval

engram evaluation data

The evaluation data behind Typed Decisions in Agent Memory: Where They Help, Where They Don't, and What It Costs (Rishabh Sharma, 2026, doi:10.5281/zenodo.22948964; version 1: doi:10.5281/zenodo.22941758): update sets that extend LoCoMo with fact changes, labeled contradiction pairs, the relation decisions escalated to an LLM, and every scored answer from the paper's runs. Code: the engram repository (bench/make_hf_dataset.py builds this directory from the repository's committed files, with no API calls).

Configs

How it was made

• —Systems. engram (an LLM extracts facts; every later decision is a typed question answered by Jev) and mem0 OSS 2.1.0 (ADD-only, default config). Extraction claude-haiku-4-5 for both, with mem0's extraction prompt and inputs; answers and judging claude-sonnet-4-6 with mem0's LoCoMo evaluation prompts, temperature 0.
• —Slices. dev: conv-26 sessions 1–4 (76 messages, 35 questions). Held-out: conv-30, conv-41, conv-42 and conv-43 whole (610 scored questions in four categories, plus 190 adversarial questions), run once after the configuration was frozen.
• —Adversarial questions have no answer in the conversation; the judge received the abstention "Not mentioned in the conversation" as gold. mem0's answer prompt does not ask for abstention; both systems share that handicap.
• —Known gaps. The token-matched mem0 run (run = mem0_token_matched__heldout_pooled__k6) saved labels and token counts but not answer text, so answer is null there. The dev-slice runs of Table 1 predate token counting, so retrieved_tokens is null there.

Bias and limitations

Update sets 1–2 were drafted and labeled with an AI assistant (Claude) at the author's direction; the author reviewed a subset. The 50 contradiction pairs were written and labeled by the author. The author also designed the system being evaluated, so the sets test the failure modes its builders anticipated, and they are small (30 + 20 items, 50 pairs). The judge is an LLM whose agreement with human labels was not measured. Scores come from one model stack (claude-haiku-4-5 extraction, claude-sonnet-4-6 answers and judge) and are not comparable to LoCoMo leaderboards run on other stacks.

LoCoMo

This dataset does not include LoCoMo's conversations, questions or gold answers; LoCoMo questions are identified by conversation and index. Get LoCoMo from snap-research/locomo (data/locomo10.json; Maharana et al., ACL 2024), which is licensed CC BY-NC 4.0 (non-commercial). Some fields here derive from LoCoMo content: update_set_1.original_fact paraphrases conv-26 messages, and per_question.answer is model output about LoCoMo conversations. This dataset is therefore released under the same license, CC BY-NC 4.0: free to share and adapt with attribution, not for commercial use.

Citation

@misc{sharma2026typed,
  author    = {Sharma, Rishabh},
  title     = {Typed Decisions in Agent Memory: Where They Help, Where They Don't, and What It Costs},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.22948964},
  url       = {https://doi.org/10.5281/zenodo.22948964},
  note      = {Version 1.1. Version 1: doi:10.5281/zenodo.22941758}
}

LoCoMo:

@inproceedings{maharana2024locomo,
  title     = {Evaluating Very Long-Term Conversational Memory of {LLM} Agents},
  author    = {Maharana, Adyasha and Lee, Dong-Ho and Tulyakov, Sergey and Bansal, Mohit and Barbieri, Francesco and Fang, Yuwei},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  pages     = {13851--13870},
  year      = {2024},
  doi       = {10.18653/v1/2024.acl-long.747}
}