nebula/GenImage-arrow

GenImage Arrow

Generator- and split-partitioned Arrow release of the GenImage benchmark. Each leaf directory is also a standalone Hugging Face save_to_disk bundle.

Contents

• —Train: 2,581,150 valid images across eight generators.
• —Test: 100,000 images across eight generators.
• —Validation is an alias of test because the official GenImage val directory is the benchmark test set. It is not a third independent split.
• —Seventeen unavailable or zero-byte upstream train entries are documented in manifest.json.

Quick start

from datasets import load_dataset

test = load_dataset("nebula/GenImage-arrow", "all-test", split="test")
train = load_dataset("nebula/GenImage-arrow", "all-train", split="train")
biggan = load_dataset("nebula/GenImage-arrow", "biggan-train", split="train")
adm_test = load_dataset("nebula/GenImage-arrow", "adm-test", split="test")
biggan_all = load_dataset("nebula/GenImage-arrow", "biggan")

Configuration names follow these rules:

• —all-train, all-test, and all-validation load every generator for one split.
• —adm, biggan, glide, midjourney, sd14, sd15, vqdm, and wukong expose all available splits for one generator.
• —Add -train, -test, or -validation to download only one split of one generator, for example sd15-test.

The *-validation configurations point to the same physical Arrow shards as their *-test counterparts and therefore do not duplicate storage.

Download a standalone Arrow bundle

Every data/<split>/<generator> directory is a complete Hugging Face Dataset.save_to_disk bundle. Download only the leaf directory you need, then open it with load_from_disk:

from pathlib import Path

from datasets import load_from_disk
from huggingface_hub import snapshot_download

snapshot = Path(
    snapshot_download(
        repo_id="nebula/GenImage-arrow",
        repo_type="dataset",
        allow_patterns="data/test/ADM/*",
    )
)
adm_test = load_from_disk(snapshot / "data/test/ADM")
print(len(adm_test))  # 12000

Change both test and ADM in the pattern to select a different physical split or generator. Exact generator directory names are:

ADM
BigGAN
glide
Midjourney
stable_diffusion_v_1_4
stable_diffusion_v_1_5
VQDM
wukong

Recommended toolbox: AID

For synthetic-image detector training and evaluation, use our iamwangyabin/AID toolbox:

git clone https://github.com/iamwangyabin/AID.git
cd AID

This dataset release includes AID compatibility indexes. Each train leaf contains train.json; each test leaf contains test.json. These are small indexes over the existing Arrow rows and do not contain duplicate image data.

Download a leaf bundle as shown above, then configure AID's existing data.ArrowDatasets loader. No AID loader code changes are required:

datasets:
  source:
    - target: data.ArrowDatasets
      data_root: /path/to/downloaded-snapshot/data/test/ADM
      sub_sets: [ADM]
      split: test
      benchmark_name: GenImage

The data_root must point to a generator leaf, not the repository root. The single value in sub_sets must exactly match that leaf's generator name. To evaluate multiple generators, add one source entry per generator and change both fields together. For training, use data/train/<generator>, its exact generator name, and split: train.

For AID, use physical split: test for the official GenImage validation/test partition. The Hub's validation split is only an alias of those same test shards, so there is intentionally no separate validation.json compatibility index.

This compatibility path applies to AID configurations that use data.ArrowDatasets. The detector named AIDE uses its own AIDEBinaryJsonDatasets preprocessing contract and is not covered by this drop-in configuration.

Schema

Every row contains image_path, embedded image bytes, binary label, and generator. Train rows additionally retain source md5, width, and height metadata. Binary labels use 0 for real images and 1 for generated images.

License and source

The original GenImage dataset is released under CC BY-NC-SA 4.0 with additional dataset terms restricting use to non-commercial purposes. See the upstream license and project for the authoritative terms:

• —https://github.com/GenImage-Dataset/GenImage/blob/main/License
• —https://github.com/GenImage-Dataset/GenImage