wAI-org/tmax-image-pool

tmax image pool

Every Apptainer/SIF image referenced by the three swerl-tmax-15k variants, stored once and shared between them.

Read the layout from the repo, not from a prefix

The plain images are split across four directories, not one. A consumer that filters on images/ silently gets 4,519 fewer files and then reports itself complete — the tasks that reference the missing blobs fail later, at sandbox start, one by one.

The batching exists only because HuggingFace enforces a hard limit of 10,000 files per directory at push time; images_b00..b02 are continuations of images/ with no other meaning. More batches may be added, so enumerate whatever directories exist rather than hard-coding this list.

Resolve paths through task_to_blob.json instead of guessing — it already carries the correct directory for every task.

task_to_blob.json

Keyed by variant, then by task id, valued by the blob path within this repo:

{
  "original":           {"task_000868_f0470dff": "images_b00/hamishi740__swerl-tmax-v3__17c682d5c609.sif", ...},
  "hardened_prefilter": {...},
  "solvable":           {...}
}

Tasks outnumber blobs because distinct tasks share an image.

Plain vs derived

images*/ are the registry images unchanged — a fresh apptainer pull of the original reference is byte-identical to the stored SIF.

derived/ are single-step derivations used by the hardened variants: an oracle file removed, and/or the dependency the verifier imports installed, with per-task pinned versions. A task in a hardened variant may map to either kind, so do not assume a variant uses only one directory.

Using it

import json
from huggingface_hub import hf_hub_download, snapshot_download

m = json.load(open(hf_hub_download("wAI-org/tmax-image-pool", "task_to_blob.json",
                                   repo_type="dataset")))
blobs = sorted(set(m["solvable"].values()))          # only what this variant needs
snapshot_download("wAI-org/tmax-image-pool", repo_type="dataset",
                  allow_patterns=blobs, local_dir="sifs/")

Then point each row's env_config.image at the absolute local .sif path; prefer_local_sif() passes an absolute path straight through with no registry lookup.

Fetching only one variant's blobs is much cheaper than the whole pool — solvable needs 6,968 of the 16,203.

Verifying what you fetched

Check apptainer inspect on each distinct SIF a variant references. Size and hash agreement is not sufficient: SIFs have been seen to match both and still not be valid squashfs. Count coverage per variant against task_to_blob.json rather than trusting a fetcher's own "done".