OpenRAL/rskill-rtdetr_v2_r50vd-any-coco-fp16

rskill-rtdetr-v2-r50vd

OpenRAL rSkill — RT-DETRv2 (Real-Time DEtection TRansformer v2) with a ResNet-50vd backbone, trained on COCO 2017. Runs as a perception producer on the camera tee and publishes ObjectsMetadata to /openral/perception/objects. No actuators. This skill uses kind: detector (ADR-0037); it emits no Action chunks and drives no ros2_control joints. Weights are a direct mirror of PekingU/rtdetr_v2_r50vd (Apache-2.0). This repo adds the OpenRAL rskill.yaml manifest.

What it does

RT-DETRv2-R50 detects 80 COCO-category objects in each camera frame and publishes per-frame ObjectsMetadata events containing bounding boxes, class labels, and confidence scores. The runtime ObjectsDetector (in openral_runner) reads the detector manifest block at configure time to initialise the inference session and bind the class-id → label mapping.

The OpenRAL detector perception path (ros_image_detector_node → DetectorRunner → ObjectsDetector) is ONNX-based, so this rSkill ships an ONNX export (model.onnx + external-data model.onnx.data) produced by tools/export_rtdetr_onnx.py. The manifest declares runtime: tensorrt: on a CUDA host the runtime_tensorrt backend builds and caches an fp16 TensorRT engine from the ONNX on first load; on hosts without the tensorrt group, onnxruntime runs the same ONNX graph (CPU or CUDA EP) as the portable fallback. The weights/ PyTorch checkpoint remains for standalone transformers inference (see Standalone inference below).

RT-DETRv2 improves over RT-DETR v1 with selective multi-scale feature extraction, a discrete sampling operator, and improved training strategies.

Supported robots / embodiments

This detector is embodiment-agnostic: it requires only an RGB camera of at least 640×480 and emits ObjectsMetadata. All known embodiment tags are declared in the manifest; the sensors_required entry sets modality: rgb with no vla_feature_key, so the loader accepts any RGB camera stream regardless of its key name.

Sensors / observation contract

The detector emits no Action chunks and has no proprioception (observation.state) contract.

Latency

Budget declared in manifest: per_chunk_ms: 50.0.

VRAM

The manifest defaults to dtype: fp16. For the <500 MB budget use fp16.

Accuracy (COCO val2017)

Weights

Two artefacts ship in this rSkill:

1. 1.ONNX (used by the OpenRAL detector path) — model.onnx + model.onnx.data. Not committed to git (binary artefact; see .gitignore). Reproduce with the same ephemeral overlay used for rtdetr-coco-r18:

   uv run --isolated --no-project \
       --with "transformers>=4.45,<5" --with "torch>=2.2" --with torchvision \
       --with onnx --with onnxscript \
       python tools/export_rtdetr_onnx.py \
       --out rskills/rtdetr-v2-r50vd/model.onnx \
       --model-id PekingU/rtdetr_v2_r50vd

Use --isolated --no-project — a plain uv run --with overlays the project venv whose torchvision is built against a different torch, breaking the RTDetrForObjectDetection import. Never uv sync --group onnx-export (it prunes pydantic/structlog from the dev venv).

The new torch exporter is not bit-reproducible across toolchain versions; treat the digests as a same-host integrity check. The published copies on the HF Hub repo are canonical.

1. 1.PyTorch (`weights/model.safetensors`) — mirrored from PekingU/rtdetr_v2_r50vd, same Apache-2.0 license, with the upstream config.json and preprocessor_config.json. Used only by the standalone transformers example below; the OpenRAL detector path does not load it.

Upstream model / training

This rSkill packages RT-DETRv2 (Real-Time DEtection TRansformer v2) with a ResNet-50vd backbone (`r50vd`). It copies no new weights — both the ONNX export and weights/model.safetensors derive from the upstream Transformers checkpoint (see the Weights section above).

Usage in OpenRAL

Activate the skill

ral skill activate OpenRAL/rskill-rtdetr-v2-r50vd

Reference in robot manifest

perception_producers:
  - skill_id: "hf://OpenRAL/rskill-rtdetr-v2-r50vd"
    role: "s1"

Standalone inference (Python)

import torch
from PIL import Image
from transformers import RTDetrV2ForObjectDetection, RTDetrImageProcessor

image_processor = RTDetrImageProcessor.from_pretrained(
    "OpenRAL/rskill-rtdetr-v2-r50vd", subfolder="weights"
)
model = RTDetrV2ForObjectDetection.from_pretrained(
    "OpenRAL/rskill-rtdetr-v2-r50vd", subfolder="weights"
).half().cuda()

image = Image.open("kitchen.jpg")
inputs = image_processor(images=image, return_tensors="pt")
inputs = {k: v.half().cuda() for k, v in inputs.items()}

with torch.no_grad():
    outputs = model(**inputs)

results = image_processor.post_process_object_detection(
    outputs,
    target_sizes=torch.tensor([(image.height, image.width)], device="cuda"),
    threshold=0.5,
)
for result in results:
    for score, label_id, box in zip(
        result["scores"], result["labels"], result["boxes"]
    ):
        label = model.config.id2label[label_id.item()]
        print(f"{label}: {score:.2f} {[round(i, 2) for i in box.tolist()]}")

Supported object classes (80 COCO)

Household objects: bottle, wine glass, cup, fork, knife, spoon, bowl, banana, apple, sandwich, orange, broccoli, carrot, hot dog, pizza, donut, cake, chair, couch, potted plant, bed, dining table, toilet, tv, laptop, mouse, remote, keyboard, cell phone, microwave, oven, toaster, sink, refrigerator, book, clock, vase, scissors, teddy bear, hair drier, toothbrush

People & animals: person, cat, dog, bird, horse, sheep, cow, elephant, bear, zebra, giraffe

Outdoor / transport: car, bicycle, motorcycle, bus, train, truck, airplane, boat, traffic light, fire hydrant, stop sign, parking meter, bench, backpack, umbrella, handbag, tie, suitcase, frisbee, skis, snowboard, sports ball, kite, baseball bat, baseball glove, skateboard, surfboard, tennis racket

Manifest summary

Full schema: `openral_core.schemas.RSkillManifest`.

Citation

@article{lv2024rtdetrv2,
  title={RT-DETRv2: Improved Baseline with Bag-of-Freebies for Real-Time Detection Transformer},
  author={Lv, Wenyu and Zhao, Yian and Chang, Qinyao and Huang, Kui and Wang, Guanzhong and Liu, Yi},
  journal={arXiv preprint arXiv:2407.17140},
  year={2024}
}

License

• —Weights (weights/): Apache-2.0, mirrored from PekingU/rtdetr_v2_r50vd
• —rSkill manifest and packaging (rskill.yaml, README.md): Apache-2.0