umd-zhou-lab/ColorBench
🎨 ColorBench 📖 Paper | 💻 GitHub ColorBench is a multimodal dataset to comprehensively assess capabilities of VLMs in color understanding, including color perception, reasoning, and robustness, introduced in "ColorBench: Can VLMs See and Understand the Colorful World? A Comprehensive Benchmark for Color Perception, Reasoning, and Robustness". It provides: More than 5,800 image-text questions covering diverse application scenarios and practical challenges for VLMs evaluation.… See the full description on the dataset page: https://huggingface.co/datasets/umd-zhou-lab/ColorBench.
🎨 ColorBench
ColorBench is a multimodal dataset to comprehensively assess capabilities of VLMs in color understanding, including color perception, reasoning, and robustness, introduced in "ColorBench: Can VLMs See and Understand the Colorful World? A Comprehensive Benchmark for Color Perception, Reasoning, and Robustness".
It provides:
- More than 5,800 image-text questions covering diverse application scenarios and practical challenges for VLMs evaluation.
- 3 categories and 11 tasks for various color-centric capabilities evaluation including Perception (Color Recognition, Color Extraction and Object Recognition), Reasoning (Color Proportion, Color Comparison, Color Counting, and more) and Robustness.
📃 Instruction
The data/test*.parquet files contain the dataset annotations and images pre-loaded for processing with HF Datasets.
from datasets import load_dataset
color_bench = load_dataset("umd-zhou-lab/ColorBench")📂 Dataset Description
The dataset contains the following fields: