wAI-org/tmax-image-pool

1---2license: other3tags:4- terminal-agent5- reinforcement-learning6- apptainer7---8 9# tmax image pool10 11Every Apptainer/SIF image referenced by the three `swerl-tmax-15k` variants, stored once12and shared between them.13 14## Read the layout from the repo, not from a prefix15 16**The plain images are split across four directories, not one.** A consumer that filters17on `images/` silently gets 4,519 fewer files and then reports itself complete — the18tasks that reference the missing blobs fail later, at sandbox start, one by one.19 20| directory | files |21|---|---:|22| `images/` | 9,971 |23| `images_b00/` | 1,933 |24| `images_b01/` | 2,000 |25| `images_b02/` | 586 |26| **plain, total** | **14,490** |27| `derived/` | 1,713 |28| **total SIFs** | **16,203** |29 30The batching exists only because HuggingFace enforces a hard limit of 10,000 files per31directory at push time; `images_b00..b02` are continuations of `images/` with no other32meaning. More batches may be added, so **enumerate whatever directories exist** rather33than hard-coding this list.34 35Resolve paths through `task_to_blob.json` instead of guessing — it already carries the36correct directory for every task.37 38## `task_to_blob.json`39 40Keyed by variant, then by task id, valued by the blob path within this repo:41 42```json43{44  "original":           {"task_000868_f0470dff": "images_b00/hamishi740__swerl-tmax-v3__17c682d5c609.sif", ...},45  "hardened_prefilter": {...},46  "solvable":           {...}47}48```49 50| variant | tasks | unresolvable |51|---|---:|---:|52| `original` — [`hamishivi/swerl-tmax-15k`](https://huggingface.co/datasets/hamishivi/swerl-tmax-15k) | 14,601 | 0 |53| `hardened_prefilter` — [`wAI-org/swerl-tmax-15k-hardened-prefilter`](https://huggingface.co/datasets/wAI-org/swerl-tmax-15k-hardened-prefilter) | 13,565 | 0 |54| `solvable` — [`wAI-org/swerl-tmax-15k-solvable-gpt-5-6-terra`](https://huggingface.co/datasets/wAI-org/swerl-tmax-15k-solvable-gpt-5-6-terra) | 7,015 | 0 |55 56Tasks outnumber blobs because distinct tasks share an image.57 58## Plain vs derived59 60`images*/` are the registry images unchanged — a fresh `apptainer pull` of the original61reference is byte-identical to the stored SIF.62 63`derived/` are single-step derivations used by the hardened variants: an oracle file64removed, and/or the dependency the verifier imports installed, with per-task pinned65versions. A task in a hardened variant may map to either kind, so do not assume a66variant uses only one directory.67 68## Using it69 70```python71import json72from huggingface_hub import hf_hub_download, snapshot_download73 74m = json.load(open(hf_hub_download("wAI-org/tmax-image-pool", "task_to_blob.json",75                                   repo_type="dataset")))76blobs = sorted(set(m["solvable"].values()))          # only what this variant needs77snapshot_download("wAI-org/tmax-image-pool", repo_type="dataset",78                  allow_patterns=blobs, local_dir="sifs/")79```80 81Then point each row's `env_config.image` at the absolute local `.sif` path;82`prefer_local_sif()` passes an absolute path straight through with no registry lookup.83 84Fetching only one variant's blobs is much cheaper than the whole pool — `solvable` needs856,968 of the 16,203.86 87## Verifying what you fetched88 89Check `apptainer inspect` on each distinct SIF a variant references. Size and hash90agreement is not sufficient: SIFs have been seen to match both and still not be valid91squashfs. Count coverage per variant against `task_to_blob.json` rather than trusting a92fetcher's own "done".93