donglixu/ORT-ImageNet-AliTok

ORT: AliTok pretokenized ImageNet-1K

Precomputed AliTok token IDs for the ImageNet-1K training split used by our ORT project. 1,281,167 image records; each record contains 10 crop variants of 273 tokens. Vocabulary size: 4096. Labels are the original integer IDs (0–999). Only label and tokens are included. The token sequences can be decoded using the matching AliTok tokenizer; they are not anonymized image representations.

This repository is publicly downloadable without an access request. Availability here does not replace the ImageNet access agreement or grant additional rights to the underlying images. Use is subject to the source terms, including non-commercial research and educational use. No MIT or Apache license is asserted for this dataset.

Usage

from datasets import load_dataset
data = load_dataset("donglixu/ORT-ImageNet-AliTok", split="train")
data.save_to_disk("data/imagenet_alitok_train")

Use this directory with the ORT training code. The trainer chooses one crop per image visit. Keep all crops grouped per image. This dataset is for AliTok-based generators, not RAR/MaskGIT (256 tokens, 1024 vocabulary).

Provenance and validation

Exported from our existing merged AliTok token dataset, preserving row order, labels and all crop token IDs. Private storage metadata and internal IDs are omitted. The exact historical image cropping pipeline and class-name mapping have not been independently reconstructed; ten crop variants are verified from the stored data. The publication script checks every record's label range, crop count, sequence length and token range. Checksums and row counts are in manifest.json once upload finishes. Until that manifest is present, the upload is incomplete.