VQA-Illusion/FashionMnist_test

IllusionFashionMNIST — Test Set

Dataset summary

This repository contains the public test split of IllusionFashionMNIST, introduced in Illusory VQA: Benchmarking and Enhancing Multimodal Models on Visual Illusions. Each metadata row identifies a Fashion-MNIST target and can be paired across five image conditions: source-condition, illusion, filtered illusion, illusionless control, and filtered illusionless control.

The source-condition images originate from Fashion-MNIST and were resized to 512 × 512 pixels. Illusion images were generated from these inputs and English scene prompts with ControlNet. The filtered variants contain the preprocessing output evaluated in the paper.

Repository structure

All five indexed directories use the same filename stem. For example, <code>FashionMnist1</code> corresponds to <code>FashionMnist1.jpg</code> in every variant directory.

Metadata schema

The CSV labels describe the target embedded in the illusion and shown in the source-condition image. For either illusionless directory, override the target with No illusion.

Label mapping

The following order matches Fashion-MNIST and the official experiment code.

Download

~~~bash pip install -U huggingface_hub pandas pillow ~~~

~~~python from huggingfacehub import snapshotdownload

datasetdir = snapshotdownload( repoid="VQA-Illusion/FashionMnisttest", repotype="dataset", ) print(datasetdir) ~~~

Command-line alternative:

~~~bash huggingface-cli download VQA-Illusion/FashionMnisttest \ --repo-type dataset \ --local-dir FashionMnisttest ~~~

Load all five image conditions

~~~python from pathlib import Path import pandas as pd from huggingfacehub import snapshotdownload

root = Path(snapshotdownload( repoid="VQA-Illusion/FashionMnisttest", repotype="dataset", )) df = pd.readcsv(root / "dfdata.csv", dtype={"label": "int64"})

folders = { "illusion": "illimages", "illusionfiltered": "illusionimagesfiltered", "illusionless": "illusionlessimages", "illusionlessfiltered": "illusionlessimagesfiltered", "raw": "raw_images", }

for condition, folder in folders.items(): df[condition + "path"] = df["imagename"].map( lambda name, folder=folder: root / folder / (name + ".jpg") )

idtolabel = { 0: "T-shirt/top", 1: "Trouser", 2: "Pullover", 3: "Dress", 4: "Coat", 5: "Sandal", 6: "Shirt", 7: "Sneaker", 8: "Bag", 9: "Ankle boot", 10: "No illusion", }

longrows = [] for row in df.itertuples(index=False): for condition in folders: targetid = 10 if condition.startswith("illusionless") else int(row.label) longrows.append({ "imagename": row.imagename, "condition": condition, "imagepath": getattr(row, condition + "path"), "labelid": targetid, "labeltext": idtolabel[target_id], })

evaluationdf = pd.DataFrame(longrows) assert evaluationdf["imagepath"].map(Path.exists).all() ~~~

Filtered variants

The paper's preprocessing pipeline applies a Gaussian blur, an averaging blur, a median blur, grayscale conversion, and sharpening. The exact OpenCV implementation and parameters are reported in Appendix K of the paper. The files in the two filtered directories are the released outputs of that pipeline.

Intended use

This split supports:

• —comparison of source-condition, illusion, and filtered-illusion accuracy;
• —evaluation of No illusion rejection using matched illusionless controls;
• —zero-shot or fine-tuned VQA/classification benchmarking; and
• —robustness studies across paired image transformations.

Recommended classification metrics are accuracy, macro precision, macro recall, and macro F1. Keep all image conditions for an <code>image_name</code> in the same evaluation partition to avoid paired-data leakage.

Dataset creation and safety

The authors generated English scene descriptions with several language models and used ControlNet to combine the prompts with Fashion-MNIST source-condition images. Human reviewers validated dataset quality. The paper reports that the public datasets were screened with NSFW detectors and that flagged images were excluded.

The paper reports 1,267 IllusionFashionMNIST test samples, whereas the current public repository contains 1,152 rows in <code>df_data.csv</code>. This card documents the repository as currently hosted; use the current metadata file for reproducible indexing.

Important usage notes

• —<code>df_data.csv</code> is the authoritative index; do not infer targets from top-level folder names.
• —Hugging Face may auto-detect the variant folders as <code>imagefolder</code> classes. Those folder-derived labels are image conditions, not Fashion-MNIST categories.
• —For illusionless variants, the correct answer is No illusion (numeric ID 10), not the class stored in the row.
• —<code>raw.jpg</code> is not part of the indexed 1,152-example evaluation set.
• —This benchmark primarily covers one large hidden category per image; consult the paper for full limitations.

License

This dataset repository declares the MIT license. Users should also review and comply with any applicable terms associated with Fashion-MNIST and other upstream components.

Citation

~~~bibtex @misc{rostamkhani2024illusoryvqa, title = {Illusory VQA: Benchmarking and Enhancing Multimodal Models on Visual Illusions}, author = {Rostamkhani, Mohammadmostafa and Ansari, Baktash and Sabzevari, Hoorieh and Rahmani, Farzan and Eetemadi, Sauleh}, year = {2024}, eprint = {2412.08169}, archivePrefix = {arXiv}, primaryClass = {cs.CV}, url = {https://arxiv.org/abs/2412.08169} } ~~~

Contact

Questions and reproducibility issues can be submitted through the official GitHub repository.