OpenRAL/rskill-rtdetr_coco_r18-any-coco-fp32

rskill-rtdetrcocor18-any-coco-fp32

OpenRAL rSkill — RT-DETR (Real-Time DEtection TRansformer) with a ResNet-18vd backbone (r18vd), trained on COCO and exported to ONNX. Runs as a perception producer on the camera tee and publishes ObjectsMetadata to /openral/perception/objects. No actuators. This skill uses kind: detector; it emits no Action chunks and drives no ros2_control joints.

What it does

RT-DETR r18vd detects 80 COCO-category objects in each camera frame and publishes per-frame ObjectsMetadata events containing bounding boxes, class labels, and confidence scores. The runtime ObjectsDetector (in openral_perception) reads the detector manifest block at configure time to initialise the ONNX inference session and bind the class-id → label mapping.

Latency

Budget declared in manifest: per_chunk_ms: 50.0.

Weights

The model.onnx file is not committed to the repository (binary artefact; see .gitignore). Reproduce it with an ephemeral overlay environment (does not mutate the project venv):

uv run --isolated --no-project \
    --with "transformers>=4.45,<5" --with "torch>=2.2" --with torchvision \
    --with onnx --with onnxscript \
    python tools/export_rtdetr_onnx.py \
    --out rskills/rtdetr-coco-r18/model.onnx \
    --model-id PekingU/rtdetr_r18vd_coco_o365

Do NOT run this via uv sync --group onnx-export — uv sync reconciles the project venv to the synced group set and prunes pydantic/structlog (and other deps) the source-on-PYTHONPATH dev/test setup relies on, breaking the unit tests. --isolated --no-project is required: a plain uv run --with overlays on the project venv, whose torchvision is built against a different torch than the overlay's — importing RTDetrForObjectDetection then dies with operator torchvision::nms does not exist. The isolated form builds a clean ephemeral env (project venv untouched). transformers<5 keeps the stable RTDetr forward (logits + pred_boxes) signature; onnx+onnxscript are required by the torch ≥2.7 ONNX exporter. GPU footprint: ~0.2 GB at 640² fp32 — runs on an 8 GB card, no quantization needed.

The torch 2.9 new exporter splits the model into two files that must be kept together in the same directory:

Full sha256 values (reproduced with transformers 4.x + torch 2.9 + onnxscript; the new torch exporter is not bit-reproducible across toolchain versions, so treat these as a same-host integrity check, not a cross-version guarantee):

• —model.onnx: bda4dbeceff130cec050e9757c9d95e217526a00730fb5f1558f960a6b316c63
• —model.onnx.data: 8dff132e55df1befdf394a672a29906e38df7653be66705c47bd2a41634567b2

The published copies on the HF Hub repo are the canonical artefacts; the local export above must match them on the same toolchain.

Upstream model / training

This rSkill packages an RT-DETR (Real-Time DEtection TRansformer) object detector with a ResNet-18vd backbone (`r18vd`), exported to ONNX. It copies no PyTorch policy weights — the ONNX graph is produced from the upstream Transformers checkpoint by tools/export_rtdetr_onnx.py (see the Weights section above for the exact command and sha256 digests).

Supported robots / embodiments

This detector is embodiment-agnostic: it consumes any RGB camera stream and emits ObjectsMetadata. All known embodiment tags are declared in the manifest; the sensors_required entry has no vla_feature_key, so the loader accepts any camera key — not just camera1.

Sensors / observation contract

The detector emits no Action chunks and has no proprioception (observation.state) contract.

Manifest summary

Full schema: `openral_core.schemas.RSkillManifest`.

License

Weights: Apache-2.0 (PaddlePaddle RT-DETR public release). See arxiv:2304.08069 for the paper.