zer0int/CLIP-Cross-Attn-MUX-Training-Data-Pack

CLIP-MUX Training Data Pack

This repository is the data/metadata companion for reproducing the CLIP-MUX / x-attention CLIP training pipeline.\ Used to train model: zer0int/CLIP-ViT-L-14-Cross-Attn-Read-NoRead-ModeMUX

This is not one monolithic dataset under one license. Each component is independently scoped and carries its own LICENSE or NOTICE file.

The pack intentionally separates:

• —assets that can be redistributed directly;
• —runtime labels/manifests derived from upstream datasets;
• —recipes for datasets that users must construct from separately obtained restricted inputs.

Contents and licenses

The repository-level license metadata is therefore other: the applicable license is determined by component path.

Redistributed assets

CLEVR subset

The CLEVR component is a selected/preprocessed derivative of the CLEVR dataset. The original CLEVR data is released under CC BY 4.0.

Changes made for this training corpus include square-crop/selection preprocessing intended to preserve depicted objects for CLIP training; samples for which an acceptable object-preserving crop could not be produced were excluded. Compact object metadata used by the training pipeline is included alongside the images.

Upstream project: <https://cs.stanford.edu/people/jcjohns/clevr/>

Handwriting overlays

These overlay assets were created for this project and are released under CC0 1.0 Universal to the extent the contributor holds copyright or related rights in them.

Salt and pepper image set

This project-specific image set is released under CC0 1.0 Universal to the extent the contributor holds copyright or related rights in the material. Some images may have been produced using generative-AI tools. CC0 applies only to rights, if any, held by the contributor; it does not create rights in material that is not protectable or grant third-party trademark, privacy, publicity, or other rights.

Upstream datasets reconstructed by the preparation script

TextCaps

CLIP-MUX uses TextCaps images and human reference captions/OCR information. The development copy was obtained through shwetkm/TextCaps-Caption-Summary; generated summary/QA fields are not part of the CLIP-MUX runtime representation. The preparation tool downloads the upstream Hugging Face dataset and materializes the image/manifests layout expected by the training loader.

TextCaps is distributed under CC BY 4.0.

• —Official TextCaps site: <https://textvqa.org/textcaps/download/>
• —Acquisition source used by this project: <https://huggingface.co/datasets/shwetkm/TextCaps-Caption-Summary>

SPRIGHT-COCO

The preparation tool obtains images from SPRIGHT-T2I/spright_coco and reconstructs the data/<shard>/<key>.jpg layout expected by the training code. This pack contains the CLIP-MUX training label variants and trusted-reading metadata, but does not duplicate the upstream image payload.

SPRIGHT-COCO is distributed under the Intel Research Use License.

Upstream dataset: <https://huggingface.co/datasets/SPRIGHT-T2I/spright_coco>

Restricted-input recipes: ImageNet

ImageNet images are not distributed in this repository. Users must obtain ImageNet independently under its applicable terms and provide a local ImageNet root to the preparation script.

The recipe manifests in restricted_recipes/ contain portable relative references and project-authored construction metadata for reproducing the digital-text and handwriting-overlay derivatives used during training. Any CC0 dedication in those folders covers only project-authored recipe contributions and does not change the terms governing ImageNet or WordNet material.

ImageNet access/terms: <https://image-net.org/>

Intended use

This pack exists to make the accompanying CLIP-MUX training code reproducible without requiring users to reconstruct project-specific preprocessing, label shortening, word-avoidance rewrites, overlay selections, or small shareable corpora by hand.

It is designed for research and model-training workflows subject to the license of each component and any independently obtained upstream datasets.

Data preparation

The Git repository provides a preparation utility that will eventually support a workflow of the form:

python prepare_training_data.py \
    --data-root /path/to/clip_mux_data \
    --imagenet-root /path/to/ILSVRC2012

The utility is responsible for downloading shareable project assets, obtaining TextCaps/SPRIGHT from their upstream locations, validating runtime metadata, and constructing ImageNet-derived datasets locally from the user's own ImageNet copy.

Attribution and provenance

Attribution required by upstream licenses is recorded in the relevant component directories. The component-scoped license/notice files take precedence over any general description in this card.

No warranty

The data and metadata are provided for reproducibility and research use without warranty. Users are responsible for complying with the terms of upstream datasets and for determining whether their intended use is permitted.

Hugging Face representation

The redistributable image payloads are stored as Parquet with embedded image bytes. The repository exposes three explicit Hugging Face dataset configurations:

from datasets import load_dataset

clevr = load_dataset("REPO_ID", "clevr")
handwriting = load_dataset("REPO_ID", "handwriting_overlays")
salt_pepper = load_dataset("REPO_ID", "salt_n_pepper")

The remaining runtime metadata and reconstruction recipes intentionally stay as ordinary JSON/JSONL/CSV/TXT repository files. They are consumed by the CLIP-MUX data-preparation utility rather than treated as independent tabular dataset configurations.

Parquet image records embed the original bytes and store only portable basenames in the image path field. Local staging paths are not part of the published representation.