anonymous-researcher-X0006/KamonBench

KamonBench

A grammar-based image-to-structure benchmark for evaluating compositional factor recovery in vision-language models, built around Japanese family crests (kamon, 家紋).

Each composite crest is paired with:

• —a formal kamon description language string (KDL, kamon yōgo, 家紋用語),
• —a segmented Japanese analysis,
• —an English translation,
• —a non-linguistic program code over the generator factors.

Because every crest is synthesized from a known triple of generator factors (container C, modifier R, motif M), KamonBench supports direct factor metrics, controlled factor-pair recombination splits, counterfactual motif- sensitivity tests under fixed (container, modifier) contexts, and linear probes of factor accessibility from frozen representations. See the accompanying paper for details and baselines.

The companion code (package, training and evaluation pipelines, and the generator) lives at `anon-submittere/KamonBench` during anonymous review.

Quick start

from huggingface_hub import snapshot_download

local_dir = snapshot_download(
    repo_id="anonymous-researcher-X0006/KamonBench",
    repo_type="dataset",
)
# Then unpack kamon_bench.zip into a `dataset01/` directory; each Croissant
# file is paired with the archive (via SHA-256) and references images at
# dataset01/*.png.

Files

The Croissant files live next to the archive (not inside it), because each file pins the archive's SHA-256.

Dataset structure

The image archive contains 54,116 PNGs under dataset01/:

Splits assign whole component groups together with their parent composite, so component records share the split of the composite they belong to.

Each Croissant record in the images record set has these fields:

For program-label experiments, the same images are paired with non-linguistic codes for the container (C:NNN), modifier (X:N), and motif (M:NNN); the three *_holdout.json Croissant variants reassign splits so that whole factor combinations ((C, M), (R, M), or (C, R, M)) are absent from training, while the underlying primitive tokens still appear individually in training.

Recombination splits

The three holdout variants share the same images as the main file but reassign the train/dev/test labels so that every test composite contains a held-out factor combination not seen during training. Primitive tokens remain represented in training, so the test isolates the question of whether a model can bind familiar primitives in novel combinations rather than recall whole crests.

Limitations and intended use

• —KamonBench is a research benchmark for compositional visual recognition, factor-aware evaluation, and representation analysis. It is not an authoritative cultural or historical catalogue of kamon.
• —The crests are synthetically rendered from upstream motif assets; they differ in style and polish from professionally rendered crests and do not cover the full distribution of historical traditions.
• —The released generator uses a limited grammar (one level of containment, a fixed set of containers and modifiers).
• —See rai:dataLimitations, rai:dataBiases, and rai:dataSocialImpact in the Croissant metadata for the formal RAI description.

License

The dataset is released under CC BY-NC 4.0; see LICENSE.txt for the full legal code. The companion code is released under the MIT License.

The component images bundled with KamonBench (one isolated motif per composite and one container per contained composite) are repackaged in PNG form from the Rebolforces kamondataset, a publicly available collection of Japanese kamon motifs originally scraped from a catalogue website that is no longer accessible online (preserved via the Internet Archive); upstream provenance cannot be tracked further. We make no copyright claim over those source images and release KamonBench solely for non-commercial research use.

Citation

@misc{kamonbench2026,
  title  = {KamonBench: A Grammar-Based Dataset for Evaluating Compositional Factor Recovery in Vision-Language Models},
  author = {Anonymous},
  year   = {2026},
  note   = {Under review at NeurIPS 2026 Evaluations \& Datasets Track},
}