Datasetpublic

neurips26/MultimodalUnlearningEvalBenchmark

🧠 Multimodal Unlearning Evaluation Benchmark 📌 Overview This dataset provides evaluation outputs for studying metric inconsistency in multimodal machine unlearning. It supports reproducibility of results in: Metric Unreliability in Multimodal Machine Unlearning (NeurIPS 2026) 📊 Contents File Description 📄 multimodal_results.json Results on VQA benchmarks (MLLMU-Bench, UnLOK-VQA, MMUBench) 📄 unimodal_results.json CIFAR-10… See the full description on the dataset page: https://huggingface.co/datasets/neurips26/MultimodalUnlearningEvalBenchmark.

sourceHugging Facecc-by-4.0updated 5mo agoView on Hugging Face

1likes83downloads

Dataset Card

🧠 Multimodal Unlearning Evaluation Benchmark

📌 Overview

This dataset provides evaluation outputs for studying metric inconsistency in multimodal machine unlearning.

It supports reproducibility of results in:

Metric Unreliability in Multimodal Machine Unlearning (NeurIPS 2026)

📊 Contents

File	Description
📄 `multimodal_results.json`	Results on VQA benchmarks (MLLMU-Bench, UnLOK-VQA, MMUBench)
📄 `unimodal_results.json`	CIFAR-10 baseline results
⚖️ `uqs_weights.json`	Learned weights for Unified Quality Score (UQS)
🏆 `ranking_table.json`	Method rankings across metrics
📈 `analysis_results.json`	Correlation and disagreement analysis
🔍 `kr_pilot_results.json`	Knowledge Recoverability (KR) pilot results
🤖 `blip2_minimal_summary.json`	Cross-architecture validation (BLIP-2)

🎯 Purpose

This benchmark evaluates five standard unlearning metrics:

—Forget Accuracy (FA)
—Retain Accuracy (RA)
—Membership Inference Attack (MIA)
—Activation Distance (AD)
—JS Divergence (JS)

⚠️ Key finding:

These metrics produce conflicting rankings and do not measure knowledge recoverability (KR).

⚙️ Usage

All results in the paper can be reproduced directly from these files.

Example:

python

import json

with open("multimodal_results.json") as f:
    data = json.load(f)


📚 Source Datasets (Not Included)

This benchmark builds on:

MLLMU-Bench
UnLOK-VQA
MMUBench
CIFAR-10

These datasets are not redistributed here. Please refer to their original sources.

⚖️ License

This dataset is released under the CC-BY-4.0 License.

⚠️ Notes
This dataset contains evaluation outputs, not raw training data
Designed for benchmarking and reproducibility
Prepared to support anonymous peer review
🔗 Citation
Anonymous. Metric Unreliability in Multimodal Machine Unlearning. NeurIPS 2026.

---