microsoft/IMAGE_UNDERSTANDING

1---2license: cdla-permissive-2.03dataset_info:4- config_name: object_detection_single5  features:6  - name: id7    dtype: int328  - name: image9    dtype: image10  - name: prompt11    dtype: string12- config_name: object_detection_pairs13  features:14  - name: id15    dtype: int3216  - name: image17    dtype: image18  - name: prompt19    dtype: string20- config_name: object_recognition_single21  features:22  - name: id23    dtype: int3224  - name: image25    dtype: image26  - name: prompt27    dtype: string28  - name: ground_truth29    dtype: string30- config_name: object_recognition_pairs31  features:32  - name: id33    dtype: int3234  - name: image35    dtype: image36  - name: prompt37    dtype: string38  - name: ground_truth39    dtype: string40- config_name: spatial_reasoning_lrtb_single41  features:42  - name: id43    dtype: int3244  - name: image45    dtype: image46  - name: prompt47    dtype: string48  - name: ground_truth49    dtype: string50- config_name: spatial_reasoning_lrtb_pairs51  features:52  - name: id53    dtype: int3254  - name: image55    dtype: image56  - name: prompt57    dtype: string58  - name: ground_truth59    dtype: string60- config_name: visual_prompting_single61  features:62  - name: id63    dtype: int3264  - name: image65    dtype: image66  - name: prompt67    dtype: string68  - name: ground_truth69    dtype: string70- config_name: visual_prompting_pairs71  features:72  - name: id73    dtype: int3274  - name: image75    dtype: image76  - name: prompt77    dtype: string78  - name: ground_truth79    dtype: string80configs:81- config_name: object_detection_single82  data_files:83  - split: val84    path: object_detection_single/object_detection_val_long_prompt.parquet85- config_name: object_detection_pairs86  data_files:87  - split: val88    path: object_detection_pairs/object_detection_val_long_prompt.parquet89- config_name: object_recognition_single90  data_files:91  - split: val92    path: spatial_reasoning_lrtb_single/recognition_val.parquet93- config_name: object_recognition_pairs94  data_files:95  - split: val96    path: spatial_reasoning_lrtb_pairs/recognition_val.parquet97- config_name: spatial_reasoning_lrtb_single98  data_files:99  - split: val100    path: spatial_reasoning_lrtb_single/spatial_reasoning_lrtb_single.parquet101- config_name: spatial_reasoning_lrtb_pairs102  data_files:103  - split: val104    path: spatial_reasoning_lrtb_pairs/spatial_reasoning_lrtb_pairs.parquet105- config_name: visual_prompting_single106  data_files:107  - split: val108    path: visual_prompting_single/visual_prompting_val.parquet109- config_name: visual_prompting_pairs110  data_files:111  - split: val112    path: visual_prompting_pairs/visual_prompting_val.parquet113---114 115A key question for understanding multimodal performance is analyzing the ability for a model to have basic 116vs. detailed understanding of images. These capabilities are needed for models to be used in117real-world tasks, such as an assistant in the physical world. While there are many dataset for object detection118and recognition, there are few that test spatial reasoning and other more targeted task such as visual prompting.119The datasets that do exist are static and publicly available, thus there is concern that current AI models could120be trained on these datasets, which makes evaluation with them unreliable. Thus we created a dataset that is121procedurally generated and synthetic, and tests spatial reasoning, visual prompting, as well as object recognition122and detection. The datasets are challenging for most AI models and by being procedurally generated the123benchmark can be regenerated ad infinitum to create new test sets to combat the effects of models being trained124on this data and the results being due to memorization.125 126This dataset has 4 sub-tasks: Object Recognition, Visual Prompting. Spatial Rea-127soning, and Object Detection. 128 129For each sub-task, the images consist of images of pasted objects on random130images. The objects are from the COCO object list and are gathered from internet data. Each object is131masked using the DeepLabV3 object detection model and then pasted on a random background from the132Places365 dataset.  The objects are pasted in one of four locations, top, left, bottom, and right, with small133amounts of random rotation, positional jitter, and scale.134 135There are 2 conditions “ single” and “ pairs”, for images with one and two objects. Each test set uses 20136sets of object classes (either 20 single objects or 20 pairs of objects), with four potential locations and four137backgrounds classes, and we sample 4 instances of object and background. This results in 1280 images per138condition and sub-task.139 140__Object Detection__141 142	Answer type: Open-ended143 144	Example for "single":145 146		{"images": ["val\\banana\\left\\fire_station\\0000075_Places365_val_00030609.jpg"], "prompt": "You are an object detection model that aims to detect all the objects in the image.\n\nDefinition of Bounding Box Coordinates:\n\nThe bounding box coordinates (a, b, c, d) represent the normalized positions of the object within the image:\n\na: The x-coordinate of the top-left corner of the bounding box, expressed as a percentage of the image width. It indicates the position from the left side of the image to the object's left boundary. The a ranges from 0.00 to 1.00 with precision of 0.01.\nb: The y-coordinate of the top-left corner of the bounding box, expressed as a percentage of the image height. It indicates the position from the top of the image to the object's top boundary. The b ranges from 0.00 to 1.00 with precision of 0.01.\nc: The x-coordinate of the bottom-right corner of the bounding box, expressed as a percentage of the image width. It indicates the position from the left side of the image to the object's right boundary. The c ranges from 0.00 to 1.00 with precision of 0.01.\nd: The y-coordinate of the bottom-right corner of the bounding box, expressed as a percentage of the image height. It indicates the position from the top of the image to the object's bottom boundary. The d ranges from 0.00 to 1.00 with precision of 0.01.\n\nThe top-left of the image has coordinates (0.00, 0.00). The bottom-right of the image has coordinates (1.00, 1.00).\n\nInstructions:\n1. Specify any particular regions of interest within the image that should be prioritized during object detection.\n2. For all the specified regions that contain the objects, generate the object's category type, bounding box coordinates, and your confidence for the prediction. The bounding box coordinates (a, b, c, d) should be as precise as possible. Do not only output rough coordinates such as (0.1, 0.2, 0.3, 0.4).\n3. If there are more than one object of the same category, output all of them.\n4. Please ensure that the bounding box coordinates are not examples. They should really reflect the position of the objects in the image.\n5.\nReport your results in this output format:\n(a, b, c, d) - category for object 1 - confidence\n(a, b, c, d) - category for object 2 - confidence\n...\n(a, b, c, d) - category for object n - confidence."}147 148	Example for "pairs":149 150		{"images": ["val\\hair drier_broccoli\\left\\church-indoor\\0000030_0000059_Places365_val_00000401.jpg"], "prompt": "You are an object detection model that aims to detect all the objects in the image.\n\nDefinition of Bounding Box Coordinates:\n\nThe bounding box coordinates (a, b, c, d) represent the normalized positions of the object within the image:\n\na: The x-coordinate of the top-left corner of the bounding box, expressed as a percentage of the image width. It indicates the position from the left side of the image to the object's left boundary. The a ranges from 0.00 to 1.00 with precision of 0.01.\nb: The y-coordinate of the top-left corner of the bounding box, expressed as a percentage of the image height. It indicates the position from the top of the image to the object's top boundary. The b ranges from 0.00 to 1.00 with precision of 0.01.\nc: The x-coordinate of the bottom-right corner of the bounding box, expressed as a percentage of the image width. It indicates the position from the left side of the image to the object's right boundary. The c ranges from 0.00 to 1.00 with precision of 0.01.\nd: The y-coordinate of the bottom-right corner of the bounding box, expressed as a percentage of the image height. It indicates the position from the top of the image to the object's bottom boundary. The d ranges from 0.00 to 1.00 with precision of 0.01.\n\nThe top-left of the image has coordinates (0.00, 0.00). The bottom-right of the image has coordinates (1.00, 1.00).\n\nInstructions:\n1. Specify any particular regions of interest within the image that should be prioritized during object detection.\n2. For all the specified regions that contain the objects, generate the object's category type, bounding box coordinates, and your confidence for the prediction. The bounding box coordinates (a, b, c, d) should be as precise as possible. Do not only output rough coordinates such as (0.1, 0.2, 0.3, 0.4).\n3. If there are more than one object of the same category, output all of them.\n4. Please ensure that the bounding box coordinates are not examples. They should really reflect the position of the objects in the image.\n5.\nReport your results in this output format:\n(a, b, c, d) - category for object 1 - confidence\n(a, b, c, d) - category for object 2 - confidence\n...\n(a, b, c, d) - category for object n - confidence."}151 152__Object Recognition__153 154	Answer type: Open-ended155 156	Example for "single"157 158		{"images": ["val\\potted plant\\left\\ruin\\0000097_Places365_val_00018147.jpg"], "prompt": "What objects are in this image?", "ground_truth": "potted plant"}159 160	Example for "pairs":161 162		{"images": ["val\\bottle_keyboard\\left\\ruin\\0000087_0000069_Places365_val_00035062.jpg"], "prompt": "What objects are in this image?", "ground_truth": "['bottle', 'keyboard']"}163 164__Spatial Reasoning__165 166	Answer type: Multiple Choice167 168	Example for "single"169 170		{"images": ["val\\potted plant\\left\\ruin\\0000097_Places365_val_00018147.jpg"], 171		"query_text": "Is the potted plant on the right, top, left, or bottom of the image?\nAnswer with one of (right, bottom, top, or left) only.", 172		"target_text": "left"}173 174	Example for "pairs"175 176		{"images": ["val\\bottle_keyboard\\left\\ruin\\0000087_0000069_Places365_val_00035062.jpg"], 177		"query_text": "Is the bottle above, below, right, or left of the keyboard in the image?\nAnswer with one of (below, right, left, or above) only.", 178		"target_text": "left"}179 180	What are the evaluation disaggregation pivots/attributes to run metrics for?181 182	Disaggregation by (group by):183 184	"single": (left, right, top, bottom)185	"pairs": (left, right, above, below) 186 187__Visual Prompting__188 189Answer type: Open-ended190 191	Example for "single"192 193		{"images": ["val\\potted plant\\left\\ruin\\0000097_Places365_val_00018147.jpg"], "prompt": "What objects are in this image?", "ground_truth": "potted plant"}194 195	Example for "pairs":196 197		{"images": ["val\\sheep_banana\\left\\landfill\\0000099_0000001_Places365_val_00031238.jpg"], "prompt": "What objects are in the red and yellow box in this image?", "ground_truth": "['sheep', 'banana']"}