Multilingual-Multimodal-NLP/AutoMemoryBench

<div align="center">

AutoMemoryBench

State-Contract Evaluation for Auditable Agent Memory

   

</div>

AutoMemoryBench evaluates whether an agent uses the right memory, and only the admissible memory, under a query-time state contract. Each executable contract partitions memory into required, admissible, and prohibited sets. Prohibited memories are typed as superseded, deleted, restricted, cross-namespace, or stale-tool.

<p align="center"> <img src="assets/query-time-admissibility.png" alt="Query-time memory admissibility" width="760"> </p> <p align="center"><em>Relevance is not enough: remembered evidence must also be allowed now.</em></p>

Benchmark at a Glance

The eight domains cover personal assistance, education and tutoring, customer support, DevOps workflows, research assistance, coding agents, multi-party collaboration, and office collaboration. Minimal single-axis interventions change exactly one admissibility dimension while preserving the surrounding case.

Benchmark Design

AutoMemoryBench compiles a chronological event graph into query-conditioned memory contracts. The same benchmark case is exposed through an interaction trace view for the system under test and a contract view used only by the evaluator. This separates task success from prohibited-memory influence without revealing the gold state to the tested system.

<p align="center"> <img src="assets/benchmark-framework.png" alt="AutoMemoryBench evaluation pipeline" width="100%"> </p> <p align="center"><em>From event graphs to executable state contracts, paired interventions, and hard-gated scoring.</em></p>

One Case, Two Views

<p align="center"> <img src="assets/two-view-evaluation.png" alt="Interaction trace and compiled state-contract views" width="900"> </p> <p align="center"><em>One benchmark case is evaluated through an interaction view and a hidden contract view.</em></p>

The interaction view contains the user turns, memory operations, tool calls, tool results, and observed answer or action. The compiled contract view records what the current query must use, may use, and must not use under the active time, role, namespace, lifecycle, and tool state.

Public Data

The Dataset Viewer exposes all 120,960 public probes through three splits: audit_subset, public_dev, and public_test. Each row exposes the query, probe type, task type, required and prohibited memory IDs, expected behavior, scoring rule, and state-contract identifier. This flattened view is intended for fast browsing and filtering; the canonical nested cases remain in the downloadable domain shards.

The complete public release is hosted directly in this repository:

Each archive contains one canonical JSON shard per domain. The original sample manifest and domain shards are under `data/sample`. The same release is versioned with the evaluator in the GitHub repository. The 30,240-probe hidden-test split is withheld to prevent overfitting.

# Download the three public split archives from Hugging Face.
hf download Multilingual-Multimodal-NLP/AutoMemoryBench \
  --repo-type dataset --include "data/full/*" --local-dir AutoMemoryBench-data

Canonical Case Fields

Evaluation

• —StrictCore is the primary hard gate. A memory-required query counts only when the task is solved, required evidence is used, and no prohibited memory influences the output.
• —PairAcc measures whether both sides of a minimal intervention pair pass StrictCore.
• —LeakRate measures how often prohibited memory appears in an answer or trace.
• —AMQ is a diagnostic composite across writing, retrieval, update, compression, task utility, safety, and efficiency; it is not the headline metric.

The canonical evaluator is implemented in `amb/benchmark/evaluation` and `amb/benchmark/metrics`.

Paper

• —AutoMemoryBench paper PDF
• —Submission-ready arXiv source package
• —SHA256 checksums

The formal Hugging Face Paper page and arXiv tag will be added after the paper receives an arXiv identifier.

Authors

Jiajun Wu, Jian Yang, Chen Li, Linzheng Chai, Ge Gao, Ensheng Shi, and Yuchi Ma.

Jiajun Wu and Jian Yang are affiliated with Beihang University. Ensheng Shi and Yuchi Ma are affiliated with Huawei Cloud Computing Technologies Co., Ltd.

Citation

@misc{automemorybench,
  title  = {AutoMemoryBench: State-Contract Evaluation for Auditable Agent Memory},
  author = {Wu, Jiajun and Yang, Jian and Li, Chen and Chai, Linzheng and
            Gao, Ge and Shi, Ensheng and Ma, Yuchi},
  year   = {2027},
  url    = {https://github.com/wuyuVerse/AutoMemoryBench}
}

License

The code and benchmark data are released under Apache-2.0. The paper license will follow the license selected for the arXiv submission.