IRVLUTD/RPX

RPX: Robot Perception X

RPX is a real-world RGB-D benchmark for measuring robot perception across scene changes. The canonical naren/all release combines the multi-object, egocentric, single-object, VQA, and tracking metadata that previously lived on separate dataset branches.

• —Code and benchmark toolkit: github.com/IRVLUTD/RPX
• —Recommended dataset revision: naren/all (pin the commit SHA printed by your download for reproducible results)
• —License: Creative Commons Attribution 4.0

<video controls muted loop playsinline width="100%" src="https://huggingface.co/datasets/IRVLUTD/RPX/resolve/main/assets/rpx-jumbotron.mp4"></video>

What is included

manifest/frames_v1.parquet is the canonical frame inventory. It contains 133,121 unique frame records and accurately marks absent Ego depth, fisheye, pose and auxiliary-mask modalities. manifest/frames_v2.parquet is retained only as historical metadata; it is incomplete and is not referenced by the active release index.

Get started

Install the benchmark package from the matching code branch:

git clone --branch naren/all https://github.com/IRVLUTD/RPX.git
cd RPX
python3 -m venv .venv
. .venv/bin/activate
python -m pip install -e './benchmark[hub,schemas]'
hf auth login

Download only the files needed for one task:

from rpx_benchmark.dataset_hub import download_for_task

result = download_for_task(
    task="segmentation",
    split="easy",
    repo_id="IRVLUTD/RPX",
    revision="naren/all",
)
print(result.local_dir, result.matched_scenes)

Equivalent CLI:

python -m rpx_benchmark.dataset_hub.cli download \
  --task segmentation --split easy \
  --repo-id IRVLUTD/RPX --revision naren/all

Do not clone or download the complete 237 GB repository for a single benchmark. The toolkit reads the task manifest and fetches the required shards.

How RPX works

Each MOS scene is captured in three phases:

The Effort-Stratified Difficulty (ESD) protocol assigns each of the 300 scene phases to easy, medium, or hard using the perception difficulty and annotation effort recorded in splits/scene_splits.json. Use the phase-level files in splits/ for MOS evaluation. Ego captures do not participate in ESD.

Task manifests under manifests/<task>/<split>.json identify samples and modalities. The benchmark downloads the corresponding archive shards, runs a reference model or user adapter, and reports task performance, robustness across phases, and compute cost. VQA uses its deterministic JSONL manifests and keeps answers inside the evaluator.

Tasks

The code package exposes ten generic task runners and a separate VQA evaluator. See the benchmark README for the supplied pretrained-model commands and environments.

Bring your own model or API

Custom models use the same loaders, splits and metrics as supplied models. Wrap a local model or remote inference service with a benchmark factory:

import rpx_benchmark as rpx
from my_model import predict_depth  # RGB ndarray -> metric-depth ndarray

model = rpx.make_numpy_depth_model(predict_depth, name="my-depth")
result, report, paths = rpx.run_monocular_depth(
    rpx.MonocularDepthRunConfig(
        model=model,
        split="easy",
        repo_id="IRVLUTD/RPX",
        revision="naren/all",
        output_dir="rpx_results/my-depth/easy",
    )
)
print(result.aggregated)

The package also provides factories and public runners for video depth, segmentation, detection, open-vocabulary detection, tracking, grounding, relative pose, sparse depth, and keypoint matching. Your wrapper owns authentication and service-specific configuration; the evaluator records predictions, errors, timing and provenance.

For VQA, provide a callable with this contract:

from rpx_benchmark.vqa import evaluate_vqa

# predict_vqa(image_paths, prompt, max_new_tokens, output_kind) -> raw text
report = evaluate_vqa(
    "vqa_manifest.jsonl",
    predict_vqa,
    model_name="my-vlm",
    model_revision="my-checkpoint-or-api-version",
    image_cache=".cache/rpx-vqa",
    output_dir="rpx_results/my-vlm",
)

Return one JSON object with label and bbox, where bbox is XYXY normalized to 0–1000; binary questions return yes or no. For two-image questions, image_paths is ordered (reference_crop, target), and the predicted box always refers to the target image. The evaluator never passes ground-truth answers or boxes to the model callable.

Identity and join rules

RPX uses separate identifiers because objects and masks have different scopes:

The source of truth is manifest/object_catalog_v1.json. Join MOS masks through manifest/mos_mask_object_map_v1.parquet; join tracking prompts through tracking/metadata/text_initialization_v1/scene_condition_vocab.parquet. Questionnaire paths and SOS paths in the object catalog resolve to the same canonical object.

Repository layout

assets/                         dataset-card media
manifest/                       canonical inventories, object catalog and checksums
manifests/<task>/<split>.json   executable benchmark sample manifests
objects/                        70 single-object scan shard trees
objects_meta/                   per-object metadata and questionnaires
preview/                        lightweight MOS and Ego previews
scenes/                         100 MOS phase trees and 100 Ego view trees
splits/                         ESD phase tables and scene-level provenance
tracking/metadata/              text-initialization vocabularies and checksums
vqa/                            full VQA Parquets and benchmark JSONL manifests

Large modalities are packed in tar shards. RGB and fisheye imagery use lossless WebP members; depth and masks use PNG; camera poses are (7,) float64 NPY vectors ordered [x, y, z, qx, qy, qz, qw]. Labels are versioned under labels/<name>/v<N>.tar.

VQA data

The runnable release manifest is vqa/metadata/benchmark_subset_v1/benchmark_available_31500.jsonl. benchmark_pending_4500.jsonl contains reserved descriptors, not runnable questions: Ego has no depth, so those spatial questions have not been fabricated. Published checksum and validation reports remain beside each VQA release.

Reproducibility and validation

• —Pin the resolved commit SHA for published results.
• —Verify downloaded files with the SHA-256 manifests under manifest/, vqa/metadata/, and tracking/metadata/.
• —Treat manifest/current.json as the active metadata index.
• —Preserve source_catalog_id as text when reading CSV; prefer JSON or Parquet when schema fidelity matters.
• —Use manifest/frames_v1.parquet for all frame discovery.

The consolidation preserves every file from main, then adds the complete VQA and tracking metadata trees from their source branches. manifest/release_naren_all_v1.json records source revisions, counts and the branch-level integrity audit.

Citation and license

If you use RPX, cite the accompanying paper. The final BibTeX entry will be added when the paper is released.

The RPX dataset is released under the Creative Commons Attribution 4.0 International (CC BY 4.0) license. You may share and adapt it with attribution. The benchmark code is distributed separately under the MIT license in the GitHub repository.