Datasetpublic

jiyounglee0523/VisAlign

VisAlign: Dataset for Measuring the Alignment between AI and Humans in Visual Perception This is the test set of VisAlign (NeurIPS 2023 Datasets and Benchmarks Track), a dataset for measuring the degree of alignment between AI models and humans in visual perception. It contains 900 images across 8 categories. Ground-truth labels and per-image categories are withheld, and filenames are anonymized IDs — to evaluate your model, submit your predictions to the VisAlign Leaderboard.… See the full description on the dataset page: https://huggingface.co/datasets/jiyounglee0523/VisAlign.

sourceHugging Facecc-by-4.0updated 1mo agoView on Hugging Face

1likes69downloads

Dataset Card

VisAlign: Dataset for Measuring the Alignment between AI and Humans in Visual Perception

This is the test set of VisAlign (NeurIPS 2023 Datasets and Benchmarks Track), a dataset for measuring the degree of alignment between AI models and humans in visual perception. It contains 900 images across 8 categories.

Ground-truth labels and per-image categories are withheld, and filenames are anonymized IDs — to evaluate your model, submit your predictions to the VisAlign Leaderboard.

—📄 Paper: arXiv:2308.01525
—🏆 Leaderboard: huggingface.co/spaces/jiyounglee0523/leaderboard
—💻 Code: github.com/jiyounglee-0523/VisAlign

Dataset Structure

Each sample has:

Column	Type	Description
`image`	`Image`	The image
`file_name`	`string`	Anonymized image ID (`va_<hex>.jpg`) — the key to use in leaderboard submissions

Categories

The test set is composed of the following categories. The category of each individual image is withheld (it is part of the evaluation and is only used server-side when scoring submissions).

Category	Group	#	Description
1	Must-Act	100	Unaltered samples of the 10 classes
2	Must-Act	100	Animals in incongruous backgrounds (generated with Stable Diffusion)
3	Must-Act	100	Category 1 samples with adversarial perturbation (FGSM)
4	Must-Abstain	100	Objects that do not belong to any of the 10 classes
5	Must-Abstain	100	Chimeras combining features of two different animals
6	Must-Abstain	100	Mammals biologically close to the 10 target mammals
7	Must-Abstain	100	Non-photorealistic styles (e.g., drawings, sculptures)
8	Uncertain	200	Images cropped at varying sizes/regions or corrupted with one of 15 corruption types (intensity 1–10)

Prediction format

For leaderboard submission, your model should output an 11-dimensional distribution per image, corresponding, in order, to:

[tiger, zebra, camel, giraffe, elephant, rhino, gorilla, bear, kangaroo, human, abstain]

The last dimension (abstain) represents "none of the 10 mammals / uncertain / unrecognizable".

Usage

python

from datasets import load_dataset

ds = load_dataset("jiyounglee0523/VisAlign", split="test")
print(ds[0]["file_name"])

Citation

bibtex

@article{lee2023visalign,
  title={Visalign: Dataset for measuring the alignment between ai and humans in visual perception},
  author={Lee, Jiyoung and Kim, Seungho and Won, Seunghyun and Lee, Joonseok and Ghassemi, Marzyeh and Thorne, James and Choi, Jaeseok and Kwon, O-Kil and Choi, Edward},
  journal={Advances in Neural Information Processing Systems},
  volume={36},
  pages={77119--77148},
  year={2023}
}

License

CC-BY-4.0