OpenRAL/rskill-smolvla-franka_panda-maniskill3-bf16

rskill-smolvla-franka_panda-maniskill3-bf16

OpenRAL rSkill — SmolVLA (0.45 B) finetuned on a 1000-demo Franka LiftCube dataset in ManiSkill3 SAPIEN, packaged for use with the OpenRAL robot agent framework.

This package wraps `Calvert0921/smolvla_franka_liftcube_1000` with a rskill.yaml manifest that adds capability checking, license surfacing, latency budgets, and local registry integration. It does not copy model weights.

What this skill does

Picks and lifts a single cube on a tabletop with a Franka Panda arm in the ManiSkill3 SAPIEN simulator. Action chunks of length 50; two RGB camera views (overhead + wrist); 9-D Franka proprio state.

Upstream model / training

The training data was collected with a pd_joint_pos Franka in SAPIEN; the checkpoint therefore outputs 8-D joint commands (7 arm + 1 gripper). State inputs are the Franka qpos (9 values: 7 arm joints + 2 finger joints) — the gym observation agent.qpos exactly matches this layout.

Observation → action contract

Supported robots / embodiments

Sensors required

Mirrors rskill.yaml::sensors_required:

Manifest summary

Full schema: `openral_core.schemas.RSkillManifest`.

Quick start

from openral_rskill.loader import rSkill

pkg = rSkill.from_yaml("rskills/smolvla-maniskill-franka/rskill.yaml")
print(pkg.manifest.name, pkg.manifest.version)
print(pkg.manifest.weights_uri)

Reproduction

# One-time bootstrap
just bootstrap && just sync --group sim --group maniskill3

# End-to-end rollout (live SAPIEN window via --view).
DISPLAY=:1 uv run --group sim --group maniskill3 \
    openral sim run --config scenes/benchmark/maniskill_pick_cube.yaml \
                --rskill rskills/smolvla-maniskill-franka \
                --view

The runner deferred-opens the SAPIEN window after the policy load (~25 s) so the window manager never sees an unresponsive empty viewer. On a warm cache the policy lifts the cube within ~200 steps (reward accumulates from 0 to ~50 over the rollout).

Evaluation

The shipped LiftCube checkpoint scores 0.0 on PickCube-v1 (task mismatch — see the LiftCube gap above); the headline will lift once a PickCube-trained policy replaces it. Re-run via the curated franka_panda suite (it auto-filters to this rSkill's task):

openral benchmark run \
    --suite maniskill3_panda \
    --vla smolvla:rskills/smolvla-maniskill-franka

How the wiring works

The model expects two RGB cameras (up / wrist), a 9-D Franka qpos state, and emits 8-D joint position commands. The end-to-end path:

1. 1.ManiSkill3 backend (python/sim/src/openral_sim/backends/maniskill3.py) surfaces every entry in sensor_data as camera1 / camera2 / ... in declaration order, plumbs backend_options.robot_uids through to gym.make (so this YAML's panda_wristcam brings in the wrist camera), and forwards task.max_steps to max_episode_steps so the rollout isn't silently truncated at MS3's default 50 steps.
2. 2.rSkill manifest declares image_preprocessing.aliases: {camera1: up, camera2: wrist} so the SmolVLA preprocessor finds what it expects, and state_contract.dim: 9 so the adapter truncates the env's qpos+qvel state to qpos-only.
3. 3.SmolVLA adapter (python/sim/src/openral_sim/policies/smolvla.py) detects the upstream model's missing policy_*processor.json files and falls back to rebuilding the lerobot processors from manifest.dataset_uri's meta/episodes_stats.jsonl — generic path that applies to any community finetune uploaded without processors.
4. 4.`--view` triggers the SAPIEN viewer_render() hook (deferred-window, mirroring PR #160's simpler_env). The window opens lazily on the first applied step, after the policy is loaded.

License

This rSkill package (rskill.yaml, README.md) is Apache-2.0. The wrapped weights at `Calvert0921/smolvla_franka_liftcube_1000` ship without an explicit LICENSE file; the base model lerobot/smolvla_base is Apache-2.0 and this derivative is treated as Apache-2.0 here on the inheritance assumption. If the upstream author later publishes a different posture, this manifest's license: field should be updated to match.

See also

• —`robots/franka_panda/robot.yaml` — RobotDescription manifest.
• —`scenes/benchmark/maniskill_pick_cube.yaml` — paired BenchmarkScene config (pass --rskill rskills/smolvla-maniskill-franka).
• —CLAUDE.md §6.4 — rSkill packaging contract.