OLAResearchX/MemoryATHENA

MemoryATHENA

This dataset repository is a compact release ledger of MemoryATHENA experiment metadata and aggregate results. It follows the release style of OLAResearchX/XMemTransfer-Results. It is not a training corpus, benchmark mirror, or redistribution of task examples, labels, or prediction logs.

What is released

• —results.jsonl: 54 compact aggregate rows (30 QA condition/task rows and 24 general-NLP condition/task rows).
• —metadata.json: release provenance, dataset references, model links, and exclusions.
• —schema.json: row schema.

The repository intentionally excludes ARTIFACTS.md, internal filesystem paths, raw datasets, labels, predictions, optimizer states, and credentials.

Training data

The model stages use causal text and a token budget. The reported training and validation counts below are processed token positions, not the number of raw documents. Downstream QA/NLP labels are never training targets.

Validation versus downstream evaluation

There are two different uses of the word “validation” in this release:

1. 1.Training validation is held-out causal text used to select memory/router checkpoints. It uses no downstream benchmark labels.
2. 2.Downstream benchmark splits are listed below. Their labels are used only for final metrics after inference, not to train memory, readers, or the router.

This distinction matters: a QA validation split is an evaluation benchmark split, whereas the 2M-position causal-text validation stream is the checkpoint-selection set.

Downstream evaluation datasets

All released QA and general-NLP results are inference-only evaluations with frozen model components. The exact split and the number of examples actually scored are listed here.

Five-task QA evaluation

Open-QA tasks report exact match (EM) and token F1. TruthfulQA reports MC1, MC2, MC3, and their arithmetic mean. NQ has 3,610 rows in the public validation split; the evaluator excludes one malformed answer-only row, so the released result scores 3,609 examples.

For the case-study and RAG diagnostics, HotpotQA uses distractor context. Supporting-fact annotations are not provided to the router or used to configure routing.

Six-task general-NLP evaluation

The primary six-task table follows the exact public kNN-Prompt task files used by the evaluator. This is important for reproducibility: MR, CR, and RT are local protocol files rather than a claim that the HF SetFit/CR or another replacement dataset is identical.

The six-task score is an unweighted mean of the six accuracies. General-NLP scoring uses domain-conditional PMI with next-token log-probability sums over label synonyms. Labels are consumed only for final accuracy.

Secondary HaluEval stress test

HaluEval is kept separate from the six-task macro-average because it uses a different binary factuality-classification protocol. The evaluator uses pminervini/HaluEval, configurations dialoguesamples, qasamples, and summarization_samples, split data, with 10,000 examples per configuration.

• —HaluEval

Yahoo threshold note

The default router evaluation uses the configured threshold tau=0. A separate Yahoo tau=0.9/1.0 sweep is a post-hoc test-set diagnostic and must not be interpreted as an independently validation-selected threshold. It does not retrain the model and is reported separately from the default evaluation.

Links

• —Project page: MemoryATHENA
• —Code: OLAResearch/ATHENA
• —Collection: MemoryATHENA
• —QA model: OLAResearchX/memoryathena-qa-20260922
• —General-NLP model: OLAResearchX/memoryathena-general-nlp-20260922
• —Reference release: OLAResearchX/XMemTransfer-Results