mahgoobi/scoring_data

scoring_data

At a state: several action chunks proposed from it, and how each one actually ended. A branch the search dropped was cut off mid-episode, so it is resumed from its own snapshot and carried to a finish — the action nobody executed still gets an answer to would this have worked.

1037 searches · 13 tasks · 39,016 nodes, each with its own state and image.

This repo hosts the data. What it means, how it was produced and how to use it live in the code that wrote it: https://github.com/EAI-RSM/rewind — see its README for the format, the search strategies, and worked examples of loading a record.

Splits

Both halves live here, told apart by the split column of meta/searches rather than by path. The blocks are disjoint by construction, so a scorer trained on one can be measured on the other without having seen it.

from rewind.record.hub import index
runs  = index("mahgoobi/scoring_data")
train = [r for r in runs if r["split"] == "train"]
test  = [r for r in runs if r["split"] == "test"]

Tasks

One row per task: how many searches it contributed, which splits, which scene configs, the seeds, and the benchmark commit whose code built those scenes.

That last column is not bookkeeping. The benchmark's success criteria change over time — put_milktea_on_shelf gained an upright requirement, put_milktea_next_to_laptop a 15° tolerance — so two runs of one task under different commits are scored by different rules and should not be pooled. Runs of a task the change did not touch stay comparable.

Contents

Outcomes, best to worst: hard_success solved it cleanly, soft_success solved it after a collision, soft_failure missed, hard_failure missed and collided. terminal says whether the episode had ended when the outcome was read — tier is an outcome only where it is true, and only those are counted here.

Loading it

from datasets import load_dataset
train = load_dataset("mahgoobi/scoring_data", "nodes", split="train")   # the collection seeds

The library's splits ARE the split column of meta/searches: the frontmatter above names each run's shard under the block that column puts it in, so the two cannot drift. Only the blocks this repo actually holds are declared.

Every node carries its own state — poses, the robot's command, and one JPEG per camera — so nodes alone answers most questions. steps is what happened between two nodes. There is no video: rewind video builds one from these frames when you want to watch a branch.

Data is partitioned as data/<table>/task=<task>/<search_id>.parquet, so one task is one directory:

from huggingface_hub import snapshot_download
snapshot_download("mahgoobi/scoring_data", repo_type="dataset",
                  allow_patterns=["meta/**", "data/*/task=drop_apple_in_bin_ks/*"])

Each run's config is stored verbatim at meta/configs/<search_id>.yml, so any run can be repeated from the record itself.

Full format, and everything else: https://github.com/EAI-RSM/rewind.