VQA-Illusion/IllusionChar_train

IllusionChar — Training Set

Dataset summary

This repository contains the training split of IllusionChar, the optical character recognition (OCR) component of Illusory VQA: Benchmarking and Enhancing Multimodal Models on Visual Illusions. The task is to transcribe a hidden, case-sensitive alphanumeric sequence from an illusory image, or return No illusion when no sequence is embedded.

Sequences contain 3–5 characters drawn from digits, uppercase Latin letters, and lowercase Latin letters. Sequence images were combined with English scene descriptions and transformed with ControlNet.

Repository structure

The value in <code>imagename</code> is the filename stem. For example, <code>illusionchar0</code> corresponds to <code>illusionimages/illusionchar0.jpg</code>.

Always index the repository through <code>dfdata.csv</code>. Do not build the training set by blindly globbing <code>illusionimages/</code>, because that would include 68 unindexed files.

Metadata schema

Use <code>keepdefaultna=False</code>; otherwise pandas converts empty No illusion labels to missing values:

~~~python import pandas as pd

metadata = pd.readcsv("dfdata.csv", keepdefaultna=False) metadata["target_text"] = metadata["label"].map( lambda value: "No illusion" if value == "" else value ) ~~~

Output encoding

IllusionChar is a sequence-transcription task, not a fixed-class classification task. Therefore, the classification ID mappings used by IllusionMNIST, IllusionFashionMNIST, and IllusionAnimals do not apply.

Do not assign one numeric class ID per complete sequence: most sequences are unique, and the official evaluation treats them as text. If a tokenizer requires character IDs, define that tokenizer explicitly and keep it separate from the dataset's ground-truth representation.

Download

~~~bash pip install -U huggingface_hub pandas pillow ~~~

~~~python from huggingfacehub import snapshotdownload

datasetdir = snapshotdownload( repoid="VQA-Illusion/IllusionChartrain", repotype="dataset", ) print(datasetdir) ~~~

Command-line alternative:

~~~bash huggingface-cli download VQA-Illusion/IllusionChartrain \ --repo-type dataset \ --local-dir IllusionChartrain ~~~

Load and pair images with metadata

~~~python from pathlib import Path import pandas as pd from huggingfacehub import snapshotdownload

root = Path(snapshotdownload( repoid="VQA-Illusion/IllusionChartrain", repotype="dataset", )) df = pd.readcsv(root / "dfdata.csv", keepdefaultna=False)

df["targettext"] = df["label"].map( lambda value: "No illusion" if value == "" else value ) df["illusionpath"] = df["imagename"].map( lambda name: root / "illusionimages" / (name + ".jpg") ) df["filteredpath"] = df["imagename"].map( lambda name: root / "illusionimagesfiltered" / (name + ".jpg") ) df["rawpath"] = df["imagename"].map( lambda name: root / "rawimages" / (name + ".jpg") ) for column in ["filteredpath", "raw_path"]: df[column] = df[column].map(lambda path: path if path.exists() else None)

assert len(df) == 9300 assert df["illusionpath"].map(Path.exists).all() assert (df["targettext"] == "No illusion").sum() == 300 ~~~

Intended use and evaluation

Suitable uses include OCR training under perceptual ambiguity, image-to-text generation, visual question answering, and robustness studies comparing source-condition, illusion, and filtered images.

Evaluate sequence-bearing examples with character error rate (CER) and word error rate (WER), as in the paper. Preserve case unless the experiment explicitly defines a case-insensitive protocol. For the No illusion subset, report detection accuracy separately or treat <code>No illusion</code> as a special exact-match output.

Filtered variants

The paper's preprocessing applies Gaussian, averaging, and median blurs followed by grayscale conversion and sharpening. Appendix K contains the exact OpenCV implementation and parameters. The released filtered directory covers the 9,000 sequence-bearing rows.

Dataset creation and safety

The authors generated 3–5 character source images, generated English scene descriptions with several language models, and applied an illusion effect with ControlNet. Human reviewers validated dataset quality. The paper reports NSFW screening and exclusion of flagged images from the public release.

The paper reports 9,900 IllusionChar training samples, while the current repository has 9,300 metadata rows. This card describes the files currently hosted; use <code>df_data.csv</code> for reproducible indexing.

Important usage notes

• —<code>df_data.csv</code> is the authoritative index.
• —Do not glob all files in <code>illusion_images/</code>; 68 files are not referenced by metadata.
• —Empty <code>label</code> values represent No illusion. Read the CSV with <code>keepdefaultna=False</code>.
• —Case is semantically meaningful.
• —Only the 9,000 sequence-bearing rows have raw and filtered counterparts.
• —The current metadata includes 10 rows with empty positive prompts; consumers should not assume every prompt field is populated.
• —The benchmark focuses on OCR under illusion rather than open-ended VQA; consult the paper for full limitations.

License

This dataset repository declares the MIT license. Users should also review and comply with any applicable terms associated with upstream models and components.

Citation

~~~bibtex @misc{rostamkhani2024illusoryvqa, title = {Illusory VQA: Benchmarking and Enhancing Multimodal Models on Visual Illusions}, author = {Rostamkhani, Mohammadmostafa and Ansari, Baktash and Sabzevari, Hoorieh and Rahmani, Farzan and Eetemadi, Sauleh}, year = {2024}, eprint = {2412.08169}, archivePrefix = {arXiv}, primaryClass = {cs.CV}, url = {https://arxiv.org/abs/2412.08169} } ~~~

Contact

Questions and reproducibility issues can be submitted through the official GitHub repository.