Datasetpublic

agentscope-ai/ReMe_longmemeval_clean_s_v2

LongMemEval ReMe Cleaned-S longmemeval_s_reme_cleaned.json is a corrected version of the LongMemEval Cleaned-S dataset. It keeps the original questions and haystack sessions while replacing the answer and supporting-session ground truth with the reviewed values from final_groundtruth_cleaned_s.json. The corrections address inaccurate answers and evidence sessions, including cases where evidence occurred after the question time and therefore leaked future information into the… See the full description on the dataset page: https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2.

sourceHugging Faceupdated 2mo agoView on Hugging Face

1likes55downloads

Dataset Card

LongMemEval ReMe Cleaned-S

longmemeval_s_reme_cleaned.json is a corrected version of the LongMemEval Cleaned-S dataset. It keeps the original questions and haystack sessions while replacing the answer and supporting-session ground truth with the reviewed values from final_groundtruth_cleaned_s.json.

The corrections address inaccurate answers and evidence sessions, including cases where evidence occurred after the question time and therefore leaked future information into the evaluation.

Sources

—Upstream dataset: xiaowu0162/longmemeval-cleaned
—Upstream file: longmemeval_s_cleaned.json
—Upstream revision: 98d7416c24c778c2fee6e6f3006e7a073259d48f
—Upstream file SHA-256: d6f21ea9d60a0d56f34a05b609c79c88a451d2ae03597821ea3d5a9678c3a442
—Reviewed ground-truth SHA-256: 97dd2b0df872f12c682041c44fe8082f9fa55ab882f5c78c9237b7c77ba9bb00
—Generated dataset SHA-256: f8b308cc2d184d029dd2226e0c0a88eeac298e6318706d588e015beb1b83fc7d

Transformation

Records are matched one-to-one by question_id. The generated file preserves the upstream schema and applies these mappings:

Output field	Value
`answer`	Reviewed `answer`
`original_answer`	Upstream `answer`
`answer_session_ids`	Reviewed `evidence_session_ids`
`original_answer_session_ids`	Upstream `answer_session_ids`

The reviewed evidence_session_ids are written to answer_session_ids so the result remains compatible with the upstream Cleaned-S format. All other fields, including question, question_date, question_type, haystack_sessions, haystack_session_ids, and haystack_dates, are retained from the upstream records.

Change statistics

Changes are counted using exact JSON value equality.

Metric	Records	Share
Total	500	100.0%
Any corrected field changed	77	15.4%
No corrected field changed	423	84.6%
`answer` changed	57	11.4%
`answer_session_ids` changed	58	11.6%
Both fields changed	38	7.6%
Only `answer` changed	19	3.8%
Only `answer_session_ids` changed	20	4.0%

Evidence-session totals changed from 948 to 819. The corrections removed 129 evidence references across 58 records and added none. Records with an empty evidence list increased from 0 to 58; all 58 corrected records with empty evidence use an answer beginning with The information provided is not enough.

Question type	Total	Answer changed	Evidence changed	Either changed
`knowledge-update`	78	2	6	8
`multi-session`	133	7	11	17
`single-session-assistant`	56	0	0	0
`single-session-preference`	30	4	0	4
`single-session-user`	70	1	0	1
`temporal-reasoning`	133	43	41	47

Validation

—Both inputs contain 500 unique question_id values, with identical ID sets.
—question_type and question time match for all 500 records.
—Every corrected evidence ID exists in the corresponding haystack_session_ids list.
—The generated file contains 500 unique records and retains every non-target value from the upstream dataset.

Citation

If you use this dataset, please cite the original LongMemEval authors:

Wu D, Wang H, Yu W, et al. Longmemeval: Benchmarking chat assistants on long-term interactive memory. arXiv preprint arXiv:2410.10813, 2024.