moorcheh/memanto-locomo-results

Memanto — LoCoMo Benchmark Results

This dataset contains the full per-question evaluation results of Memanto on the LoCoMo long-term conversational memory benchmark, covering all five stages of a progressive ablation study. It accompanies the paper:

Memanto: Typed Semantic Memory with Information-Theoretic Retrieval for Long-Horizon Agents Seyed Moein Abtahi, Rasa Rahnema, Hetkumar Patel, Neel Patel, Majid Fekri, Tara Khani Moorcheh AI / EdgeAI Innovations [arXiv preprint] · memanto.ai · moorcheh.ai

About Memanto

Memanto is a universal memory layer for agentic AI that achieves state-of-the-art accuracy on LoCoMo (87.1%) using a vector-only architecture — no knowledge graphs, no LLM-mediated ingestion, and no multi-query retrieval pipelines. It is built on Moorcheh's Information-Theoretic Search (ITS) engine, which provides deterministic semantic retrieval with sub-90ms latency and zero ingestion delay.

Key architectural features:

• —13-category typed semantic memory schema (fact, preference, decision, commitment, goal, event, instruction, relationship, context, learning, observation, error, artifact)
• —Automated conflict resolution for contradiction detection and versioning
• —Single retrieval query per question — no parallel or recursive strategies
• —Zero ingestion cost — no indexing delay, no LLM extraction at write time

About LoCoMo

LoCoMo (Long-Form Conversational Memory) is a multi-session dialogue benchmark spanning four reasoning categories:

Individual dialogues extend to 35 sessions, 300 turns, and ~9K tokens on average, making it a rigorous test of long-horizon conversational memory.

Dataset Structure

Each split corresponds to one stage of Memanto's five-stage progressive ablation study. Every row is a single evaluated question with the following fields:

Ablation Splits

The five splits map directly to the five stages of the ablation study reported in the paper:

Key finding: Stage 2 (recall expansion, k=10→40) delivers the largest single gain (+6.6 pp), confirming that retrieval recall — not architectural complexity — is the dominant performance driver.

Final Results by Category (Stage 5)

Comparison with Other Systems

Memanto achieves the highest accuracy among all vector-only systems, surpassing Mem0 by +20.2 percentage points. Hindsight attains higher accuracy but requires maximum architectural complexity (complexity score 4/4): graph DB + LLM at ingestion + multi-query retrieval + recursive querying.

How to Load in Python

from datasets import load_dataset

# Load a specific ablation stage
ds = load_dataset("moorcheh/memanto-locomo-results", split="ablation_5")
df = ds.to_pandas()
print(df.head())

# Compare accuracy across all ablation stages
for stage in range(1, 6):
    ds = load_dataset("moorcheh/memanto-locomo-results", split=f"ablation_{stage}")
    df = ds.to_pandas()
    acc = df["score"].mean() * 100
    print(f"ablation_{stage}: {acc:.1f}%")

Evaluation Setup

• —Benchmark: LoCoMo standard split, 1,540 questions
• —LLM Judge: Claude Sonnet 4 (all stages)
• —Inference Model: Claude Sonnet 4 (Stages 1–4), Gemini 3 (Stage 5)
• —Evaluation prompts: Adapted from the Hindsight repository to mitigate answerer refusal and rigid judge rejection of semantically correct but lexically divergent answers
• —Memanto version: 2.1.4
• —Retrieval backend: Moorcheh ITS engine (single query, no multi-query or recursive strategies)

Additional Assets

An Excel workbook (memanto_locomo_ablation_results.xlsx) containing all five ablation results as separate sheets with formatted headers is included in this repository for convenient offline exploration.

Reproducibility

All evaluation results in this dataset are fully reproducible. The `moorcheh-ai/memanto-evaluation` repository contains everything needed to replicate the LoCoMo experiments:

📁 [`memanto-evaluation/locomo`](https://github.com/moorcheh-ai/memanto-evaluation/tree/main/locomo) — LoCoMo evaluation directory containing:

• —📊 The benchmark dataset and question suite
• —📝 Per-ablation result files (all 5 stages)
• —🐍 Python scripts to run the full evaluation pipeline

Citation

If you use this dataset or build on Memanto, please cite:

@misc{abtahi2026memantotypedsemanticmemory,
  title         = {Memanto: Typed Semantic Memory with Information-Theoretic Retrieval for Long-Horizon Agents},
  author        = {Seyed Moein Abtahi and Rasa Rahnema and Hetkumar Patel and Neel Patel and Majid Fekri and Tara Khani},
  year          = {2026},
  eprint        = {2604.22085},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2604.22085}
}

Also cite the LoCoMo benchmark:

@misc{maharana2024evaluatinglongtermconversationalmemory,
      title={Evaluating Very Long-Term Conversational Memory of LLM Agents}, 
      author={Adyasha Maharana and Dong-Ho Lee and Sergey Tulyakov and Mohit Bansal and Francesco Barbieri and Yuwei Fang},
      year={2024},
      eprint={2402.17753},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2402.17753}, 
}

And the Moorcheh ITS engine:

@misc{abtahi2025hnswinformationtheoreticbinarizationrethinking,
      title={From HNSW to Information-Theoretic Binarization: Rethinking the Architecture of Scalable Vector Search}, 
      author={Seyed Moein Abtahi and Majid Fekri and Tara Khani and Akramul Azim},
      year={2025},
      eprint={2601.11557},
      archivePrefix={arXiv},
      primaryClass={cs.DB},
      url={https://arxiv.org/abs/2601.11557}, 
}

For questions, visit [memanto.ai](https://memanto.ai) or [moorcheh.ai](https://moorcheh.ai).