curtis-sun/SemTPCH

SemTPCH

The data archive for SemTPCH, a multimodal semantic-query benchmark of 22 analytical queries over four datasets (e-commerce, face, video, audio). Each query mixes traditional relational operators with semantic operators (semantic filter / classify / map / join / cluster / rank / aggregate) that require perception (vision, audio, or text understanding).

This repository ships only the curated benchmark data (media files + tables). Query definitions, the Palimpzest and Claude-Code runners, and the scorer live in the code repository (SemTPCH).

Quick start

# from the root of the SemTPCH code repository
tar --zstd -xf semtpch-data.tar.zst

This reconstructs build/{ecommerce,face,ava,vggsound}/, which is exactly where the runners and scorer read from:

build/
├── ecommerce/   { media/, visible.csv, full.csv }
├── face/        { media/, visible.csv, full.csv }
├── ava/         { media/, visible.csv, full.csv, action_list.csv }
└── vggsound/    { media/, visible.csv, full.csv, modalities/keyframes/ }

What's in each table

• —`visible.csv` — the input rows the system sees (500 rows per dataset). Contains the visible columns plus the media path.
• —`full.csv` — same rows plus the hidden perception columns (e.g. gender, masterCategory, articleType, hair_color, baseColour, action_ids, label). Hidden from the system; used only to build gold answers and score.
• —Numeric/order columns (list_price, discount, return_flag, order_date, …) are synthetic, generated in the style of TPC-H.

Media

For VGGSound, visible.csv has a keyframe_paths column pointing into modalities/keyframes/; the reference pipeline dispatches each row as caption → keyframe → video (keyframes are preferred when available to save cost).

File integrity

Verify after download:

echo "8b81012a47c811e5ae21c359598bda981fdeea4c0a67d74b0830ab77e5d0c309  semtpch-data.tar.zst" | sha256sum -c
zstd -t semtpch-data.tar.zst   # integrity check

The archive contains 2,661 files. It does not contain: the original (full) source datasets, raw/intermediate videos, sample manifests, download scripts, backups, or any logs — only the curated subset above.

Licensing & intended use

The media in this archive is derived from four third-party datasets. Each imposes its own (research / non-commercial) license, and redistribution of the original source datasets is not permitted by their terms. This archive is a small, curated benchmark subset provided solely so that published results on SemTPCH can be reproduced.

By downloading you agree to use this data solely for non-commercial academic research and to respect each source dataset's license. No rights to the underlying media are granted or implied beyond what the original sources allow.

Provenance notes

• —AVA clips are 10-second segments aligned to AVA's middle-frame timestamp; multiple person/action annotations at the same (video_id, timestamp) are aggregated into a single action_ids set.
• —VGGSound clips were fetched from YouTube via yt-dlp and trimmed to 10 s; keyframes were extracted for the perception dispatch above. YouTube availability of the original clips is not guaranteed.
• —The synthetic relational columns are generated in the TPC-H tradition and carry no third-party restrictions.