qruisjtu/EmbodiedRestore

EmbodiedRestore

Paired robotic first-frame observations (low-quality / ground-truth) under 25 distortions from the TID2013 / KADID-10k taxonomy, evaluated by three policies (π0.5, π0, OpenVLA). Built for benchmarking image restoration / IQA on robot-observation distributions, with downstream policy success rates(SR) and steps to successas(StS) as secondary signals.

To promote the development of image restoration model for robot vision systems, we will continue to maintain this dataset and release more first-frame observations generated by our benchmarked image restoration models, along with their corresponding SR&StS data.

A full Croissant 1.0 + RAI metadata document is shipped at the repo root: `croissant.json`. It is richer than the auto-generated Croissant served by Hugging Face — please prefer it when ingesting the dataset programmatically.

At a glance

Repository layout

EmbodiedRestore/
├── README.md
├── croissant.json                 Croissant 1.0 + RAI metadata
├── manifest.csv                   30,000 rows — one per PNG frame
├── results.csv                    7,500 rows — one per rollout
├── distort_taxonomy.csv           25 rows — id → name / family / reference
├── build_metadata.py              regenerates the four files above
├── Pi05_DistortOnly/
│   ├── distort_0/
│   │   ├── firstframe_distort_0_eps1_0.png   ... eps100_3.png   (400 PNGs)
│   │   └── result.json                        success_rate + 100 env_metadata
│   └── distort_1/ ... distort_24/
├── Pi0_DistortOnly/                same shape as above
└── openvla_DistortOnly/            same shape as above

File naming

<policy>_DistortOnly/distort_<id>/firstframe_distort_<id>_eps<e>_<v>.png

• —<id>: 0–24, see distort_taxonomy.csv
• —<e>: 1–100, episode index (1-indexed). The corresponding entry in the matching result.json is env_metadata[e-1] (0-indexed).
• —<v>: view × role suffix
• —0 = base camera, LQ (distorted)
• —1 = wrist camera, LQ
• —2 = base camera, GT (clean)
• —3 = wrist camera, GT

results.csv already normalises eps to the 1-indexed filename convention, so you can join it with manifest.csv directly:

import pandas as pd
m = pd.read_csv("manifest.csv")
r = pd.read_csv("results.csv")
joined = m.merge(r, on=["policy", "distort_id", "eps", "distort_name"])

Tabular schemas

manifest.csv (30,000 rows)

results.csv (7,500 rows)

distort_taxonomy.csv (25 rows)

id, name, family, reference — distortion families follow the TID2013 / KADID-10k benchmark taxonomy.

Quick start

from huggingface_hub import snapshot_download
local_dir = snapshot_download(
    repo_id="qruisjtu/EmbodiedRestore",
    repo_type="dataset",
)

import pandas as pd
m = pd.read_csv(f"{local_dir}/manifest.csv")

# all LQ base-camera frames under JPEG compression for OpenVLA
sel = m.query("policy == 'openvla' and distort_name == 'jpeg_compression' "
              "and camera == 'base' and role == 'lq'")
print(sel.head())

Pair LQ ↔ GT for an image-restoration training loop:

lq = m.query("role == 'lq'")
gt = m.query("role == 'gt'").rename(columns={"path": "gt_path"})
pairs = lq.merge(gt, on=["policy", "distort_id", "distort_name", "eps", "camera"]) \
          .rename(columns={"path": "lq_path"})

Croissant

import mlcroissant as mlc
ds = mlc.Dataset(jsonld="croissant.json")
for r in ds.records("rollouts"):
    print(r); break

The frames and rollouts RecordSets are the recommended entry points. For raw per-distortion JSON outputs, see the policy_results FileSet (one result.json per <policy>_DistortOnly/distort_<id>/).

Intended use

• —Primary: benchmarking image restoration and image-quality-assessment algorithms on the distribution of robot first-frame observations.
• —Secondary: studying how restoration / denoising affects downstream manipulation success across modern VLA policies.

Limitations

• —Only first-frame observations are kept.
• —Single simulator (Robosuite/MuJoCo).
• —Each distortion is applied at a fixed strength.

Ethics

Fully synthetic; contains no humans, faces, voices, or personally identifiable information.

Citation

TODO: arXiv / NeurIPS bibtex

Acknowledgements

Built on top of Robosuite and MuJoCo. Distortion taxonomy follows TID2013 and KADID-10k. Policies evaluated: π0.5, π0, OpenVLA.