ubr-physical-ai/isaac-sdg-rescue-target

isaac-sdg-rescue-target

Synthetic training data for detecting a person lying on the ground wearing a hi-vis vest, the target class of the search-and-rescue spotter. Rendered with NVIDIA Isaac Sim 6.0.1 Replicator (path-traced RTX) from posed, vested worker characters placed in real Isaac environments, with per-frame randomization of camera position, HDRI sky, key light, target pose, and distractor objects with random materials. Ground truth is exact: boxes come from the renderer, not from annotators.

Built 2026-09-10. 133760 images (120384 train / 13376 validation), 1024x768 / 1280x720 px (see width/height per image), JPEG quality 92.

Classes

distractor boxes (if present) mark random primitive shapes scattered as clutter; keep them as a negative class or drop them at training time.

Positives are the lying classes only. person_lying_vest (hi-vis vest) and person_lying (no vest: police, medical, business and other characters) are what a casualty detector must find; the upright classes are hard negatives it must not fire on. Every person class is split by hi-vis vest, so the vest's contribution to recall is measurable: person_standing_vest / person_standing, person_walking_vest / person_walking, and the same for bending, kneeling, crouching and seated. Train with the lying classes as the positive and the upright ones unlabelled (background), or as extra classes the deployment ignores; do not merge every person* box into one class, that turns the negatives into positives.

Occluded frames are negatives, not empty labels. person_visible: false on an image means the person was placed but is hidden (racking, walls, furniture); such frames carry no annotation, their pose is none, and the pose that was placed is kept as pose_placed. Frames of the negative sweep have no person at all and no pose_placed. Every image also carries distance_m and bearing_deg (camera to target), hdri, seed, resolution and camera_view.

view / camera_view values: orbit (2.2-3.4 m, 47-degree lens), roverview (4-25 m), roverview_far (10-17.5 m), roverview_far2 (17.5-30 m), night (environment lights at 3 %, night skies, a lamp riding on the camera), fog (distance fog from 3 m, dense by 45 m); every rover view uses the rover's 90-degree lens at 0.55-0.65 m. Environments: warehouse, office, hospital, the 76 x 73 m ArchVis hall, and Rivermark, an outdoor street block whose ground sits at z = 5.75 m.

Coverage

orbit: a camera ring 2.2-3.5 m out at 0.8-1.7 m height, 47-degree lens. roverview: the rover's own camera geometry, 0.58 m height, 90-degree lens, 4-8.5 m; roverview_far: the same lens at 10-17.5 m.

Vest colours: mixed, none, orange, yellow.

Format

Hugging Face imagefolder: each split folder holds JPEG images and a metadata.jsonl with one row per image:

{"file_name": "...jpg", "image_id": 0, "width": 1280, "height": 720,
 "environment": "warehouse", "pose": "supine", "vest": "yellow", "source_run": "...",
 "objects": {"id": [..], "bbox": [[x, y, w, h], ..], "category": [..], "category_name": [..], "area": [..]}}

bbox is COCO [x, y, width, height] in pixels. category indexes classes.json. view is the camera geometry the frame was rendered with (see Coverage). Merged COCO files for detector frameworks are in annotations/coco_train.json and annotations/coco_validation.json. Exact generator configs are in provenance/.

from datasets import load_dataset
ds = load_dataset("imagefolder", data_dir=".")   # or the Hub repo id
ex = ds["train"][0]; ex["image"], ex["objects"]

How it was generated

• —Generator: sdg_generate.py (config-driven Isaac Sim Replicator pipeline), verified per run by sdg_verify.py (labels drawn back onto pixels; environment labels stripped; duplicates and sub-2 px boxes removed).
• —Target asset: an Isaac People construction-worker character posed lying down on its own skeleton and baked to static meshes (assets/rescue/make_lying_pose.py); supine and prone.
• —Camera: a ring of positions [25.0, 30.0, 35.0, 40.0] m out, [0.58] m up, 24 angles; 14 HDRI skies; randomized key light.

Provenance

Caveats

• —Synthetic. Validate on real photographs from the deployed robots before trusting a number.
• —Every pose is authored, not physically fallen or captured, and comes from one character per vest colour: supine and prone for the target, six upright poses for the hard negatives.
• —Environment labels were stripped; shelving, pallets and walls are unlabeled background.
• —Rendered content derives from NVIDIA Isaac Sim assets; see License.

License

• —Annotations, metadata, configs, provenance and this card: CC BY 4.0 (UB Robotics / teex). Cite the dataset if you use it.
• —Images: rendered with NVIDIA Isaac Sim Replicator from NVIDIA-distributed assets (characters, environments, skies). NVIDIA's Isaac Sim Additional Software and Materials License (v. June 9, 2025) forbids redistributing the assets themselves and does not address rendered output; the source assets are not included here, only renders. If you are NVIDIA and read this differently, contact us and we will adjust.
• —Synthetic throughout: no real people, no real locations, no personal data.

Benchmark protocol

This dataset is the training side of a casualty-search benchmark for ground robots. The evaluation protocol we adopt (SubT-style mission scoring, per-distance detector recall, false-alarm rate on negatives, N and spread reported for every number) is in `BENCHMARK.md`, together with the standards it borrows from.

Layout on this revision (v3)

The images are packed into WebDataset-style tar shards, not loose files: 133,760 JPEGs in train-000.tar .. train-032.tar and validation-000.tar .. validation-003.tar. Loose files drew 10,630 rate-limit responses from the Hub and never finished uploading; the same content as 37 shards uploads in one pass.

import datasets
# the shards, with the per-image metadata alongside
ds = datasets.load_dataset("webdataset", data_files={"train": "train-*.tar", "validation": "validation-*.tar"})

train-metadata.jsonl and validation-metadata.jsonl carry one row per image: file_name, environment, pose, vest, person_visible, distance_m, bearing_deg, hdri, seed, resolution, and the objects block. annotations/coco_{train,validation}.json are the merged COCO files, with RLE segmentation on every annotation.

clips/ holds 109 camera orbits (93 frames each) with instance-id masks and a per-clip COCO, indexed by clips.jsonl at the root. They are the input side of a video-to-video translation step; see the clips card for the details.

Other revisions: main is the 7,440-image release in loose-file layout; v2 is a 23,960-image staging snapshot in the same sharded layout as this one.