harvardairobotics/reLAIONet

Dataset Card: reLAIONet

Dataset Summary

reLAIONet is a manually proofread, web-sourced image classification benchmark aligned to ImageNet's 1,000-class label space. Sourced entirely from open web crawls (reLAION-400M) rather than Flickr, it provides a challenging out-of-distribution complement to ImageNet val and ImageNetV2 for evaluating discriminative and class-conditional generative models.

This dataset was introduced in: Fair Benchmarking of Emerging One-Step Generative Models Against Multistep Diffusion and Flow Models.

Dataset Details

Dataset Description

• —Curated by: Advaith Ravishankar, Serena Liu, Mingyang Wang, Todd Zhou, Jeffrey Zhou, Arnav Sharma, Ziling Hu, Léopold Das, Abdulaziz Sobirov, Faizaan Siddique, Freddy Yu, Seungjoo Baek, Yan Luo, Mengyu Wang
• —Paper: arXiv:2603.14186
• —Image source: reLAION-400M (Schuhmann et al.)

Dataset Structure

Images are stored under images/<synset>/<synset>_XXXX.png. Label metadata is provided in metadata_imagenet.json, which maps each file path to its ImageNet label fields:

{
  "images/abacus/abacus_0002.png": {
    "imagenet_class_idx": 397,
    "wnid": "n02666196",
    "synset": "abacus"
  }
}

Data Fields

Construction Pipeline

1. 1.Synset matching — All 48 reLAION-400M parquet files are scanned for captions matching WordNet lemmas unique to a single ImageNet synset; shared lemmas are excluded.
2. 2.NSFW filtering — Entries flagged as NSFW are removed.
3. 3.Multi-label filtering — Captions matching more than one ImageNet class are discarded.
4. 4.CLIP similarity filtering — Pairs are filtered using CLIP ViT-B/32 cosine similarity > 0.82 against the synset description (threshold from the original LAIONet paper).
5. 5.Ranked download — Up to 70 images per class are downloaded, ranked by CLIP similarity (highest first).
6. 6.Manual proofreading — All images are hand-reviewed to remove mislabeled, visually ambiguous, or low-quality samples.

Relationship to Prior Work

reLAIONet is the only publicly available ImageNet-compatible evaluation set sourced entirely from open web crawls with manual proofreading.

Uses

Direct Use

• —Out-of-distribution generalization assessment for models trained on ImageNet
• —Class-conditional generative model evaluation (diffusion, flow-matching, GANs) requiring integer class-ID conditioning
• —Comparative benchmarking alongside ImageNet val and ImageNetV2

Out-of-Scope Use

This dataset is intended for evaluation only. Training on reLAIONet would undermine its purpose as an out-of-distribution benchmark. Image use is subject to the terms of reLAION-400M and the licenses of individual source images.

Limitations

• —Class coverage — 243 ImageNet classes have no images, typically rare biological species or specialized objects too infrequent or ambiguous in web crawl data to pass all filters.
• —Imbalanced class sizes — Classes range from 1 to 69 images depending on reLAION-400M availability and filter attrition.
• —Web crawl biases — reLAIONet inherits geographic and cultural biases present in reLAION-400M.

Citation

BibTeX:

@misc{ravishankar2026fairbenchmarkingemergingonestep,
  title={Fair Benchmarking of Emerging One-Step Generative Models Against Multistep Diffusion and Flow Models},
  author={Advaith Ravishankar and Serena Liu and Mingyang Wang and Todd Zhou and Jeffrey Zhou and Arnav Sharma and Ziling Hu and Léopold Das and Abdulaziz Sobirov and Faizaan Siddique and Freddy Yu and Seungjoo Baek and Yan Luo and Mengyu Wang},
  year={2026},
  eprint={2603.14186},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2603.14186}
}

APA:

Ravishankar, A., Liu, S., Wang, M., Zhou, T., Zhou, J., Sharma, A., Hu, Z., Das, L., Sobirov, A., Siddique, F., Yu, F., Baek, S., Luo, Y., & Wang, M. (2026). Fair Benchmarking of Emerging One-Step Generative Models Against Multistep Diffusion and Flow Models. arXiv preprint arXiv:2603.14186.