deem-data/ArtiFact

ArtiFact

ArtiFact is a large-scale multimodal benchmark of museum artwork records with aligned images and structured metadata. It is designed for evaluating metadata extraction, error detection, semantic querying, and multimodal reasoning over cultural-heritage collections.

The dataset combines records from the Rijksmuseum, the Metropolitan Museum of Art (Met), and the Art Institute of Chicago (AIC), with normalized fields for artists, dates, materials, techniques, dimensions, culture, location, and descriptions.

<img src="fig1_overview.png" alt="" style="display: inline-block; height: auto; width: auto; vertical-align: text-bottom; margin: 0 0.0rem;" />

Dataset structure

The Hub repository is organized as follows:

data/
├── ArtiFact_clean.csv
├── ArtiFact_clean_dirty.csv
├── ArtiFact_clean_dirty_sample_10.csv
├── ArtiFact_clean_dirty_sample_200.csv
├── ArtiFact_clean_dirty_sample_500.csv
└── images/
    ├── ArtiFact_clean.tar              # all clean-split images
    ├── ArtiFact_clean_dirty.tar        # all dirty-split images
    └── ArtiFact_clean_dirty/           # loose files: sample subset only
        ├── 000441.jpg
        └── ...

Splits

`ArtiFact_clean` holds normalized metadata without deliberate corruption. Use it as a training pool, reference data, or source of ground-truth labels.

`ArtiFact_clean_dirty` is the evaluation split. Roughly half of eligible rows carry one injected error; the other half are clean control rows with no injection. See How the dirty file works below.

How the ArtiFact_clean_dirty file works

Each row in ArtiFact_clean_dirty.csv contains both the correct metadata and (when applicable) the corrupted version used for benchmarking.

Example — row with a place error:

Example — clean control row:

For image swaps, the correct image URL is in image_url and the swapped URL is in image_url_error. The on-disk file for the corrupted image may use an -e suffix (e.g. 000002-e.jpg); see image_object_id_error for the source object.

Typical evaluation workflow:

1. 1.Present the model with the corrupted view — read from *_error columns when non-empty, otherwise from the base column.
2. 2.Ask whether an error exists and which field is wrong.
3. 3.Score against error_type / error_subtype and the base * columns as ground truth.

import pandas as pd

row = pd.read_csv("data/ArtiFact_clean_dirty.csv").iloc[0]

has_error = bool(str(row["error_type"]).strip())
print("injected error?", has_error)
print("category:", row["error_type"], "subtype:", row["error_subtype"])
print("correct location:", row["location"])
print("corrupted location:", row["location_error"])

ArtiFact_clean.csv has no error_type, error_subtype, or *_error columns — only ground-truth metadata.

Sample index files

For quick experiments without loading the full dirty split, we provide row-index samples of ArtiFact_clean_dirty.csv:

Each sample file is a single-column CSV of `row_id` values: the 1-based row index into ArtiFact_clean_dirty.csv (header excluded). Rows are drawn uniformly at random within each error category, restricted to records that have a local image on disk.

The corresponding images are available as loose files under data/images/ArtiFact_clean_dirty/ on the Hub (see Images). You do not need to download the full ArtiFact_clean_dirty.tar to run sample benchmarks.

To materialize a sample as a full metadata table:

import pandas as pd

dirty = pd.read_csv("data/ArtiFact_clean_dirty.csv")
sample_ids = pd.read_csv("data/ArtiFact_clean_dirty_sample_500.csv")["row_id"]
subset = dirty.iloc[sample_ids - 1].reset_index(drop=True)

You can generate new samples with the project script scripts/sample_benchmark_by_error.py (see 11_sample_benchmark_by_error.sh).

Source museums (by object_ID prefix)

Images

Images are provided in two forms:

Loose files let you run quick experiments on ArtiFact_clean_dirty_sample_*.csv without downloading the full archives. For the full CSVs or the clean split, extract the corresponding tar.

Quick start (samples only)

If you only need a sample benchmark, download the metadata CSVs, a sample index file, and the loose images under data/images/ArtiFact_clean_dirty/:

huggingface-cli download deem-data/ArtiFact \
  --repo-type dataset \
  --include "data/ArtiFact_clean_dirty.csv" \
  --include "data/ArtiFact_clean_dirty_sample_200.csv" \
  --include "data/images/ArtiFact_clean_dirty/*"

Then resolve images via image_path as usual (paths point at data/images/ArtiFact_clean_dirty/NNNNNN.jpg).

We provide three evaluation samples (2000, 4000, and 10000 records) with balanced clean and error-injected records across all 19 subtypes (100, 200, and 500 per subtype, respectively); our baseline uses the 4000-record sample.

Full dataset (tar archives)

Extract both archives so that paths match the image_path column in the full CSVs:

mkdir -p data/images
tar -xf data/images/ArtiFact_clean.tar -C data/images/
tar -xf data/images/ArtiFact_clean_dirty.tar -C data/images/

This yields:

data/images/ArtiFact_clean/000001.jpg
data/images/ArtiFact_clean_dirty/000001.jpg

ArtiFact_clean contains only clean records, while ArtiFact_clean_dirty contains an equal split of clean and erroneous records with ground truth labels.

Image files are named by row index (NNNNNN.jpg). Rows with image-swap errors may use a companion file such as 000002-e.jpg.

Download with the Hugging Face CLI

pip install -U huggingface_hub
huggingface-cli download deem-data/ArtiFact --repo-type dataset --local-dir ArtiFact
cd ArtiFact
mkdir -p data/images
tar -xf data/images/ArtiFact_clean.tar -C data/images/
tar -xf data/images/ArtiFact_clean_dirty.tar -C data/images/

Data fields

Shared metadata columns

Additional columns in ArtiFact_clean_dirty.csv

These columns exist only in the dirty split where 50% rows have an injected error. See How the dirty file works for how base vs *_error columns relate.

Base column ↔ error column pairs (correct value on the left, corrupted on the right):

Injected error subtypes

Errors are drawn from a fixed distribution over metadata and image fields, including:

• —Dates: century_shift
• —Dimensions: scale_error, aspect_swap
• —Materials: material_anachronism, material_interchange
• —Techniques: technique_anachronism, technique_interchange
• —Artists: artist_tier_1_easy … artist_tier_4_hardest
• —Culture / place: culture_tight_swap, culture_continent_swap, country_level_swap, city_level_swap
• —Images: image_tier_1_easy, image_tier_2_medium, image_tier_3_hard, embedding_swap

Usage example

import pandas as pd
from pathlib import Path

root = Path("ArtiFact")
clean = pd.read_csv(root / "data" / "ArtiFact_clean.csv")
dirty = pd.read_csv(root / "data" / "ArtiFact_clean_dirty.csv")

# Resolve an on-disk image from image_path
sample = dirty.iloc[0]
img = root / sample["image_path"]
print(sample["object_ID"], sample["error_type"], sample["error_subtype"], img.exists())

Notes

• —Museum terms: Underlying images and metadata originate from public museum APIs and web collections. Users must comply with each institution's terms of use and attribution requirements when redistributing or publishing results.
• —Archive layout: For full evaluation, images must be extracted from the .tar files before all image_path values resolve on disk.
• —Error injection: The dirty split contains synthetic errors for benchmarking; it does not reflect real cataloging mistakes at the source museums.
• —Images: Not every row has a local image; check image_path and file existence before image-based evaluation.

Citation

If you use ArtiFact in your work, please cite the dataset and acknowledge the source museums (Rijksmuseum, Metropolitan Museum of Art, Art Institute of Chicago).

@article{duarte2026artifact,
  title     = {{ArtiFact}: A Large-Scale Multi-Modal Cultural Heritage Dataset},
  author    = {Duarte, Luciano and Ovcharenko, Olga and Schelter, Sebastian},
  year      = {2026},
  url       = {https://github.com/OlgaOvcharenko/ArtiFact}
}

Contact

Repository: deem-data/ArtiFact Website: https://olgaovcharenko.github.io/ArtiFact/