koalab/all-some-vl

Language-only All vs. Some

Dataset Description

This dataset consists of a list of 840 questions that test whether models correctly interpret the universal quantifier "all" as applying to a scenario where every object has a certain property and the indefinite quantifier "some" as applying to a scenario where a non-empty subsert of all objects have a certain property. All questions in this dataset present scenarios in the form of images. Each scenario contains some number objects, of which all, some, or none have a given attribute. Each question is assigned a code the form Quantifier 1/Quantifier 2/Quantifier 3. Here, Quantifier 1 refers to the amount of objects that have some property (that may or may not be the property the question asks about), Quantifier 2 refers to the quantifier the question is asking about, and Quantifier 3 refers to the number of objects that have the property the question is asking about. For example, if there is a scenario with 5 blocks, all of which are blue, and the question is "Are all blocks orange?", then this question would be coded as All/All/None. For this dataset, an initial image is generated via Gemini 2.5 Flash Image, and following prompts modify this initial image. A group of rows in the dataset that consist of initial image generation prompts (e.g. "create an image with 6 red birds") immediately followed by prompts that modify this initial image (e.g. "modify this image so that 3 of the birds are now blue") are referred to below as a series of rows. The dataset is formatted as follows:

• —image: presents the scenario that is asked about in the question in prompt
• —attribute_all: the attribute that is given to all objects in the image when prompting Gemini 2.5 Flash Image to generate the initial image
• —prompt: the question that asks about a given quantifier in relation to the scenario presented in image
• —prompt_color: the color that is asked about in prompt
• —image_generation_prompt: the prompt given to Gemini 2.5 Flash Image to generate the image stored in image. Either used to generate the inital image or to modify a previously generated image
• —condition: stores the code for each question of the form Quantifier 1/Quantifier 2/Quantifier 3
• —colors: list of colors referred to for the questions in a given series of rows

If you use this dataset, please cite us using the following bibtex:

@article{padmanabhan2026bears,
  title={Bears, all bears, and some bears. Language Constraints on Language Models' Inductive Inferences},
  author={Padmanabhan, Sriram and Song, Siyuan and Misra, Kanishka},
  journal={arXiv preprint arXiv:2601.09852},
  year={2026}
}

Dataset Card Authors

Siyuan Song, Sriram Padmanabhan, Kanishka Misra

Dataset Card Contact

siyuansong@utexas.edu, srirampadmanabhan@utexas.edu, kmisra@utexas.edu