review-artifacts/locus-bench

LOCUS-Bench (anonymized release for peer review)

A benchmark for embodied multi-robot task planning that grades two difficulties separately. Axis S (state judgment): S0 no judgment; S1 whether a single named target is already in place; S2 which of 2 to 3 conditional candidates are absent; S3 which of 3 to 6 quantified instances are unsatisfied; S4 whether invisible implies absent under occlusion, with single-frame fallback planning. Axis M (mechanical structure): M0 none; M1 an opening-prerequisite chain; M2 independent subtasks decoupled across robots; M3 coupled collaboration; M4 a forced relay chain with a pushable carrier. 21 of the 25 cells are populated; ten task families instantiate them (49 family-cells).

Scenes are rendered with the official VIKI renderer (ManiSkill3 over RoboCasa kitchens and VIKI-Assets objects) with six heterogeneous robots and a 34-class object vocabulary (20 classes from VIKI, 14 new). Every reference plan replays through the same official simulator that scores predictions. Samples with an empty reference plan (73 across the evaluation splits) are ones where the correct output is no action at all. No evaluation data flows into training.

Layout

<split>/data.parquet        one row per sample: prompt, images (PNG bytes),
                            reward_model.ground_truth (task, robots, init_pos,
                            goal/temporal constraints, reference time_steps,
                            locus_meta: cell, family, split flags, seeds)
<split>/images/NNNNN.png    the same frame at native 1920x1080 (the detector
                            reads this file; the VLM reads the parquet bytes)
<split>/configs/*.yaml      scene instantiation configs (renderer input)
<split>/gt_labels/NNNNN.json detector boxes exported from instance masks (LabelMe)
<split>/render_manifest.json
MANIFEST.json               sha256 of every parquet file

Row i of data.parquet corresponds to images/{i:05d}.png, configs/*_{i:05d}.yaml and gt_labels/{i:05d}.json. The train parquet is written in small row groups; read it with pyarrow.parquet.ParquetFile(...).iter_batches() or datasets.load_dataset.

Scoring and code

Scoring reuses the official VIKI-L2 scorer plus an empty-plan extension. The scorer, the evaluation harness and the generator (task families, grid instantiation, rendering, validation) are in the anonymized code repository linked from the paper (nsdl/benchmark_scoring.py: compute_locus_l2_acc, locus_bench/generator/). Place this dataset at data/lnsdl_bench/ramp63k/ in that repository.

License and provenance

Released for peer review only; do not redistribute during review. The final license will be stated on de-anonymization (planned: CC BY 4.0 for the rendered images and annotations). Upstream components: VIKI-R / VIKI-Bench renderer and data pipeline (Apache-2.0); RoboCasa scenes (assets CC BY 4.0); VIKI-Assets objects, some of which originate from PartNet-Mobility (non-commercial research terms); ManiSkill3 (Apache-2.0). No 3D asset is redistributed here; obtain them from the respective sources under their own terms. Author, affiliation and citation information is intentionally omitted.