czty/MMV-dataset

MMV-dataset

Reproducible research splits and full-training subsets derived from long-video benchmarks. No video, audio, or subtitle media is included.

Hosted data

The LongVideoBench hidden-label test split is not used. Media references are metadata only; obtain media through each upstream project's authorized process.

Video-MME

Video-MME is part of the split design (810 train / 1,890 validation questions, grouped by 270 / 630 videos), but its official terms prohibit redistribution without prior approval. Consequently, this repository publishes only aggregate audit statistics and the deterministic materialization script—not Video-MME questions, answers, IDs, or media. After obtaining an authorized local copy, run:

python scripts/materialize_splits.py \
  --longvideobench-val path/to/longvideobench_val_v1.1.json \
  --egotempo-openqa path/to/egotempo_openQA.json \
  --videomme-parquet path/to/test-00000-of-00001.parquet \
  --output-dir local_materialized \
  --include-videomme-records

Reproducibility and leakage control

• —Fixed seed: MMV-dataset-v1-2026-09-04.
• —Video-MME, LongVideoBench, and EgoTempo are split at the video level.
• —Those selected whole groups exactly total 30% of question rows for each source.
• —Completed benchmark additions follow the explicitly documented exceptions below.
• —audit/split_report.json records counts, distributions, and zero group overlap.
• —audit/source_fingerprints.json pins source revisions and input SHA-256 values.

The hosted JSONL files require no special loader. Local Video-MME materialization requires Python 3.10+ and pyarrow (pip install -r requirements.txt).

Completed benchmark additions

The unified supervised splits now contain 1,084 train and 2,529 validation rows. EgoMemReason is exposed only through unlabeled_train and unlabeled_validation; it is not silently mixed into supervised data and no model-generated pseudo-labels are presented as truth.

M3 uses the complete 1,276-question, 100-video M3-Bench-robot annotation set. The 383/893 split is made by video, so no robot video appears in both partitions.

EgoLifeQA and EgoMemReason both use the single A1_JAKE week-long recording. Their question IDs are disjoint across 30/70 splits, but media-level isolation is mathematically impossible. Treat their held-out scores as question-held-out diagnostics, not unseen-video generalization. EgoMemReason official labels remain private and official benchmark use is contaminated once its public questions are used for training.

Evaluation warning

These are custom research splits. Training on any portion of Video-MME, EgoTempo, M3-Bench, EgoLifeQA, or EgoMemReason contaminates those official benchmarks, so scores on their remaining 70% must be reported as MMV custom held-out scores, not official full-benchmark or zero-shot leaderboard results.

Sources and attribution

• —Video-MME, official custom academic terms.
• —LongVideoBench, CC BY-NC-SA 4.0 annotations. LongVideoBench does not convey rights to linked media.
• —EgoTempo, CC BY 4.0 annotations. Ego4D media is governed separately.
• —M3-Bench, CC BY-NC-SA 4.0 annotations.
• —EgoLife, source card declares MIT; media terms apply separately.
• —EgoMemReason, CC BY-NC 4.0 public questions; answers are private.

This adaptation is non-commercial and distributed under the applicable upstream terms. The repository-level license: other reflects the mixed licensing; every hosted row carries its component license.