Datasetpublic

JadeHuang/m3eval-source-memory-supplement

M3Eval Source Memory Supplement This public dataset repo mirrors the non-archive contents of PKU-VaLuE-Lab/m3eval and adds a curated subset of source-memory videos directly under the original videos/ paths. Included video subset: 46 interleaved source-memory videos 29 split-screen dynamic-swap videos (swap10) 17 split-screen static videos (swap0) 92 total uploaded videos Notes: Original archive files are intentionally omitted from this supplement repo. selected_videos.csv and… See the full description on the dataset page: https://huggingface.co/datasets/JadeHuang/m3eval-source-memory-supplement.

sourceHugging Faceupdated 4mo agoView on Hugging Face

0likes46downloads

Dataset Card

M3Eval Source Memory Supplement

This public dataset repo mirrors the non-archive contents of PKU-VaLuE-Lab/m3eval and adds a curated subset of source-memory videos directly under the original videos/ paths.

Included video subset:

—46 interleaved source-memory videos
—29 split-screen dynamic-swap videos (swap10)
—17 split-screen static videos (swap0)
—92 total uploaded videos

Notes:

—Original archive files are intentionally omitted from this supplement repo.
—selected_videos.csv and selection_summary.json document the exact subset uploaded here.

pretty_name: M³Eval language:

—en size_categories:
—1K<n<10K task_categories:
—visual-question-answering
—multiple-choice tags:
—video
—multimodal
—benchmark
—memory
—long-video configs:
—configname: default datafiles:
—split: full path: viewer/full_questions.csv --- <h1 align="center">M³Eval: Multi-Modal Memory Evaluation through Cognitively-Grounded Video Tasks</h1>

<a href="https://jie-1203.github.io/">Jie Huang</a>1,&emsp; <a href="https://liuruixun.github.io/">Ruixun Liu</a>1,&emsp; <a href="https://github.com/PKU-VaLuE-Lab/m3eval">Sirui Sun</a>1&emsp; <a href="https://yyxxyy574.github.io/xinyiyang.github.io/">Xinyi Yang</a>1&emsp; <a href="https://yinli-vision.github.io/">Yin Li</a>2&emsp; <a href="https://yzhu.io/">Yixin Zhu</a>1&emsp; <a href="https://yiwuzhong.notion.site/homepage">Yiwu Zhong</a>1,†

1Peking University&emsp; 2University of Wisconsin-Madison

—Equal contribution.&emsp;† Corresponding author.

News

—2026-05-21: We released the M³Eval benchmark, code, and project page.

M³Eval Overview

Abstract

As multi-modal models advance towards long-form video understanding, memory emerges as a critical capability. Despite substantial effort in developing video datasets and benchmarks, existing work primarily focuses on perception and reasoning, without systematically evaluating memory: what models retain, how faithfully information is preserved, and how robust memory remains under interference.

To address this gap, we introduce M³Eval, the first comprehensive evaluation framework and benchmark for probing different memory dimensions in multi-modal models. Grounded in cognitive psychology, our design features carefully constructed tasks isolating key aspects of memory. Leveraging M³Eval, we conduct extensive experiments across representative multi-modal models, revealing consistent weaknesses and distinctive behaviors.

We find that models struggle to maintain disentangled representations when processing parallel video streams, exhibit interference patterns differing substantially from those observed in human memory, ground memory sources more reliably in the spatial domain than the temporal domain, and demonstrate limited symbolic memory. Collectively, our benchmark provides a valuable resource for future research, whereas our findings highlight memory as a fundamental yet underexplored capability and offer insights for designing more effective memory mechanisms in multi-modal models.

Main Results

Divided Attention

[image]

Accuracy (%) on three divided attention metrics under the split-screen setting without swaps and with frequent left/right swaps.

Memory Interference

[image]

Proactive: first video (V1) interferes with recall of the second video (V2); retroactive: second video (V2) interferes with recall of the first video (V1). Delta denotes proactive minus retroactive.

Interleaved Events

[image]

Accuracy (%) on four interleaved reconstruction metrics.

N-Back

[image]

Average accuracy under two symbolic attributes, scene and action, averaged over all K and N configurations.

Usage

Install

bash

git clone https://github.com/PKU-VaLuE-Lab/m3eval.git
cd m3eval/lmms-eval
uv pip install -e ".[all]"
cd ..

Download the Dataset

The benchmark is composed from existing public benchmarks. Users must follow the original licenses and usage terms of each source dataset.

Download the dataset and unpack it into data/m3eval/:

bash

huggingface-cli download PKU-VaLuE-Lab/m3eval \
  --repo-type dataset \
  --local-dir data/m3eval

bash data/m3eval/unpack_archives.sh

After unpacking, data/m3eval/ should contain:

text

data/m3eval/
├── qa_root/
├── questions/
├── nback/
├── viewer/
└── videos/
    ├── interleaved/
    ├── memory_interference/
    ├── split_screen/
    └── nback/

Evaluation

The main public entry point is:

bash

bash lmms-eval/scripts/run_m3eval_vllm.sh \
  --model_path /path/to/your/model \
  --task m3eval \
  --gpus 0 \
  --batch_size 1

For multi-GPU data-parallel evaluation, use:

bash

bash lmms-eval/scripts/run_m3eval_sharded_vllm.sh \
  --model_path /path/to/your/model \
  --task m3eval \
  --gpus 0,1,2,3 \
  --num_processes 4 \
  --batch_size 1

For a quick smoke run:

bash

bash lmms-eval/scripts/run_m3eval_vllm.sh \
  --model_path /path/to/your/model \
  --task m3eval_memory_interference \
  --gpus 0 \
  --limit 1 \
  --max_frame_num 8

Useful task names:

—m3eval
—m3eval_memory_interference
—m3eval_split_screen
—m3eval_interleaved
—m3eval_nback

The script writes outputs to lmms-eval/output/m3eval/ by default. See lmms-eval/README_M3EVAL.md for the dataset layout and task names.

To convert lmms-eval outputs into paper-facing tables:

bash

python lmms-eval/scripts/aggregate_m3eval_results.py \
  lmms-eval/output/m3eval/path/to/*_results.json

Dataset Examples

Divided Attention

[image]

Simultaneous memory for two side-by-side videos.

<details> <summary>Click to expand more examples</summary>

Memory Interference

[image]

Interference between sequentially presented videos.

Interleaved Events

[image]

Memory reconstruction from temporally interleaved clips.

N-Back

[image]

Decide whether the final clip matches the clip N positions earlier.

</details>

Citation

If you use M³Eval in your work, please cite:

bibtex

@article{huang2026m3eval,
  title   = {M3Eval: Multi-Modal Memory Evaluation through Cognitively-Grounded Video Tasks},
  author  = {Huang, Jie and Liu, Ruixun and Sun, Sirui and Yang, Xinyi and Li, Yin and Zhu, Yixin and Zhong, Yiwu},
  journal = {arXiv preprint arXiv:XXXX.XXXXX},
  year    = {2026}
}