spawlonka/bongard-rwr-plus-l3

Bongard RWR+/L3

This is a variant of the dataset featuring three images per side. The original dataset, which includes six images per side, can be found here.

Bongard Problems (BPs) provide a challenging testbed for abstract visual reasoning (AVR), requiring models to identify visual concepts from just a few examples and describe them in natural language. Early BP benchmarks featured synthetic black-and-white drawings, which might not fully capture the complexity of real-world scenes. Subsequent BP datasets employed real-world images, albeit %with real-world the represented concepts are identifiable from high-level image features, reducing the task complexity. Differently, the recently released Bongard-RWR dataset aimed at representing abstract concepts formulated in the original BPs using fine-grained real-world images. Its manual construction, however, limited the dataset size to just 60 instances, constraining evaluation robustness. In this work, we introduce Bongard-RWR+, a BP dataset composed of 5,400 instances that represent original BP abstract concepts using real-world-like images generated via a vision language model (VLM) pipeline. Building on Bongard-RWR, we employ Pixtral-12B to describe manually curated images and generate new descriptions aligned with the underlying concepts, use Flux.1-dev to synthesize images from these descriptions, and manually verify that the generated images faithfully reflect the intended concepts. We evaluate state-of-the-art VLMs across diverse BP formulations, including binary and multiclass classification, as well as textual answer generation. Our findings reveal that while VLMs can recognize coarse-grained visual concepts, they consistently struggle with discerning fine-grained concepts, highlighting limitations in their reasoning capabilities.

<div align="center"> <img src="wholes/52.jpeg" alt="Bongard-RWR+ Example" style="max-width: 50%;"> </div>

Field Descriptions

Release Date

15.05.2025

Licensing

Bongard-RWR+ is released under the CC BY 4.0 License. Please cite the corresponding paper if you use this dataset in your research.

Citation

@misc{pawlonka2025bongardrwrrealworldrepresentationsfinegrained,
      title={Bongard-RWR+: Real-World Representations of Fine-Grained Concepts in Bongard Problems}, 
      author={Szymon Pawlonka and Mikołaj Małkiński and Jacek Mańdziuk},
      year={2025},
      eprint={2508.12026},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2508.12026}, 
}