VQA-Illusion/MNIST_train

IllusionMNIST — Training Set

Dataset summary

This repository contains the training split of IllusionMNIST, introduced in Illusory VQA: Benchmarking and Enhancing Multimodal Models on Visual Illusions. The dataset is intended for training models to recognize MNIST digits embedded as visual illusions (pareidolia) in generated scenes and to reject images that contain no illusion.

MNIST source-condition images were sampled and resized to 512 × 512 pixels, combined with English scene descriptions, and transformed with a ControlNet variant. The split is balanced across digits 0–9 and an additional No illusion class.

Repository structure

The <code>imagename</code> value is the filename stem. For example, <code>Mnist1</code> corresponds to <code>illimages/Mnist1.jpg</code>.

Metadata schema

Preserve <code>label</code> as a string when reading the mixed-type column:

~~~python import pandas as pd

metadata = pd.readcsv( "dfdata.csv", dtype={"label": "string"}, keepdefaultna=False, ) metadata["label_normalized"] = metadata["label"].str.strip().str.casefold() ~~~

Label mapping

This order matches MNIST and the official experiment code.

Use ID 10 when converting the textual No illusion target to a numeric label space.

Download

~~~bash pip install -U huggingface_hub pandas pillow ~~~

~~~python from huggingfacehub import snapshotdownload

datasetdir = snapshotdownload( repoid="VQA-Illusion/MNISTtrain", repotype="dataset", ) print(datasetdir) ~~~

Command-line alternative:

~~~bash huggingface-cli download VQA-Illusion/MNISTtrain \ --repo-type dataset \ --local-dir MNISTtrain ~~~

Load and pair images with metadata

~~~python from pathlib import Path import pandas as pd from huggingfacehub import snapshotdownload

root = Path(snapshotdownload( repoid="VQA-Illusion/MNISTtrain", repotype="dataset", )) df = pd.readcsv( root / "dfdata.csv", dtype={"label": "string"}, keepdefaultna=False, )

df["illusionpath"] = df["imagename"].map( lambda name: root / "illimages" / (name + ".jpg") ) df["rawpath"] = df["imagename"].map( lambda name: root / "rawimages" / (name + ".jpg") ) df["rawpath"] = df["rawpath"].map( lambda path: path if path.exists() else None )

def tolabelid(value): value = str(value).strip() return 10 if value.casefold() == "no illusion" else int(value)

df["labelid"] = df["label"].map(tolabelid) df["labeltext"] = df["labelid"].map( lambda labelid: "No illusion" if labelid == 10 else "digit " + str(labelid) ) assert df["illusion_path"].map(Path.exists).all() ~~~

Intended use

Suitable uses include supervised illusion-aware digit classification, VQA with a constrained answer vocabulary, paired source/illusion analysis, and multimodal robustness research. For illusion classification, the expected answers are digit 0 through digit 9 and No illusion.

Dataset creation and safety

The authors generated English scene descriptions with several language models and used a ControlNet variant to combine those descriptions with resized MNIST source-condition images. Human reviewers validated dataset quality. The paper reports that the public release was screened with NSFW detectors and that flagged images were excluded.

Important usage notes

• —Use <code>df_data.csv</code> as the authoritative index.
• —Hugging Face may automatically treat top-level image directories as <code>imagefolder</code> classes. Those folder-derived labels are storage variants, not digit labels.
• —The No illusion label is textual while digit labels are numeric strings; normalize it explicitly.
• —Only illusion-bearing rows have files in <code>raw_images/</code>.
• —PyTorch pickle files can execute code during deserialization. Load <code>Mnistbalancedtrainset_indices.pth</code> only if you trust its source and need the original sampling indices.
• —The benchmark primarily contains one large hidden category per image; consult the paper for full limitations.

License

This dataset repository declares the MIT license. Users should also review and comply with any applicable terms associated with MNIST and other upstream components.

Citation

~~~bibtex @misc{rostamkhani2024illusoryvqa, title = {Illusory VQA: Benchmarking and Enhancing Multimodal Models on Visual Illusions}, author = {Rostamkhani, Mohammadmostafa and Ansari, Baktash and Sabzevari, Hoorieh and Rahmani, Farzan and Eetemadi, Sauleh}, year = {2024}, eprint = {2412.08169}, archivePrefix = {arXiv}, primaryClass = {cs.CV}, url = {https://arxiv.org/abs/2412.08169} } ~~~

Contact

Questions and reproducibility issues can be submitted through the official GitHub repository.