patrickamadeus/vlms-are-confused-tourists
Vision Language Models are Confused Tourists ✈️ 🤔 [!NOTE] We are still in the process of beautifying the README of the HF dataset. Nonetheless, our data is fully usable! Although the cultural dimension has been one of the key aspects in evaluating Vision-Language Models (VLMs), their ability to remain stable across diverse cultural inputs remains largely untested, despite being crucial to support diversity and multicultural societies. Existing evaluations often rely on… See the full description on the dataset page: https://huggingface.co/datasets/patrickamadeus/vlms-are-confused-tourists.
<div align="center"> <h1>Vision Language Models are Confused Tourists ✈️ 🤔</h1> </div>
<div align="center">
   
</div>
[!NOTE] We are still in the process of beautifying the README of the HF dataset. Nonetheless, our data is fully usable!
Although the cultural dimension has been one of the key aspects in evaluating Vision-Language Models (VLMs), their ability to remain stable across diverse cultural inputs remains largely untested, despite being crucial to support diversity and multicultural societies. Existing evaluations often rely on benchmarks featuring only a singular cultural concept per image, overlooking scenarios where multiple, potentially unrelated cultural cues coexist. To address this gap, we introduce ConfusedTourist, a novel cultural adversarial robustness suite designed to assess VLMs' stability against perturbed geographical cues. Our experiments reveal a critical vulnerability, where accuracy drops heavily under simple image-stacking perturbations and even worsens with its image-generation-based variant. Interpretability analyses further show that these failures stem from systematic attention shifts toward distracting cues, diverting the model from its intended focus. These findings highlight a critical challenge: visual cultural concept mixing can substantially impair even state-of-the-art VLMs, underscoring the urgent need for more culturally robust multimodal understanding.
📊 Dataset Specification
ConfusedTourist assesses VLM stability against perturbed geographical cues across three cultural domains:
- Cuisine 🍛
- Attire 🥋
- Musical Instruments 🪕
All perturbations are done by embedding an adversarial country's:
- FLAG 🏳️ into the scene
- A background LANDMARK 🗼
- BOTH 🗼🏳️
🚀 Usage
<details> <summary><b>📂 Option 1: Download Dataset (Recommended)</b></summary>
📥 Download the complete dataset from Hugging Face with full original + perturbed images and other data points:
Please run the following script:
from datasets import load_dataset
# Load the full suite
ds = load_dataset("patrickamadeus/vlms-are-confused-tourists")
# Access a sample
print(ds['train'][0]['item']) # Output: 'Adobo'
print(ds['train'][0]['origin_country']) # Output: 'Philippines'This is the fastest way to get started. Alternatively, you can also directly download the parquet files:
- Go to our parquet files
- Download ready-to-use features with their corresponding metadatas </details>
<details> <summary><b>🖼️ Option 2: Regenerate Image</b></summary> Visit our Github for more details. </details>
⭐️ Evaluate
Simply use our features with your designated prompt, or visit our Github for more details.
📖 Citation
@article{irawan2025vision,
title={Vision Language Models are Confused Tourists},
author={Irawan, Patrick Amadeus and Hanif, Ikhlasul Akmal and Kautsar, Muhammad Dehan Al and Winata, Genta Indra and Koto, Fajri and Aji, Alham Fikri},
journal={arXiv preprint arXiv:2511.17004},
year={2025}
}🙏 Acknowledgement
We would like to thank VLMs are Biased and VLMs are Blind for their immense inspiration in kickstarting our project in a more niche domain.