gtgando/smolvla-so101-pick-place-10cm-v3-dual

SmolVLA SO-101 Pick-and-Place (10cm, dual-camera)

`lerobot/smolvla_base` fine-tuned on 75 teleoperated demonstrations of a cube pick-and-place task with an SO-101 follower arm.

Task instruction: "Pick up the cube and place it in the bowl"

• —Code: ggand0/vla-so101
• —Dataset: gtgando/so101_pick_place_10cm_v3
• —Demo video

Results

60–80% success rate on the real robot, typical across multiple 5-episode rollouts at the 20k-step checkpoint.

Success = cube grasped and released inside the bowl within a 10s episode.

Head-to-head against the alternatives, all trained on the same dataset and evaluated at 20k with 5 rollouts each:

That 100% is a single favourable 5-episode run, not a contradiction of the 60–80% figure — treat 60–80% as the honest expectation and note that 5-episode samples are noisy. The overhead camera appears to supply global spatial context that helps the placing phase.

Training

Vision encoder frozen, action expert trained (freeze_vision_encoder=true, train_expert_only=true).

Observation / action spec

Both cameras are required — this is the dual-camera variant.

Chunk size 50, n_action_steps 50, n_obs_steps 1. Images are resized with padding to 512x512 internally. State and action use MEAN_STD normalization; visual input is IDENTITY (dataset stats are baked into the checkpoint, so no external normalization file is needed).

Usage

from lerobot.policies.smolvla.modeling_smolvla import SmolVLAPolicy

policy = SmolVLAPolicy.from_pretrained("gtgando/smolvla-so101-pick-place-10cm-v3-dual")
policy.to("cuda")

# batch requires both camera keys, observation.state, and task
action = policy.select_action(batch)

Inference loop used for the reported numbers: `scripts/infer.py`.

Training used a LeRobot fork (adds a --resume-from convenience flag and TensorBoard logging), but the checkpoint is a standard LeRobot SmolVLA checkpoint and loads with upstream LeRobot.

Limitations

• —Real-robot only. Trained and evaluated entirely on physical SO-101 hardware. Zero-shot transfer to a simulator is untested — camera intrinsics, extrinsics, lighting, and cube/bowl appearance all differ, and this policy has no domain randomization.
• —Camera placement is baked in. The overhead and wrist views are fixed to the rig described above. Moving either camera meaningfully degrades performance.
• —Fixed workspace. Demonstrations covered a ~10cm cube placement range on a single table setup.
• —Color-specific visual features. Trained on a red cube only. Tested unchanged on other colors (prompt not modified): orange, blue, and green each grasped successfully but failed or nearly failed during the place phase — 1/3 success vs 5/5 on red. Grasping generalizes; the place trajectory does not. This is the clearest evidence that the policy keys on appearance rather than an abstract "cube" concept, and it is the main reason to expect trouble on any setup whose object appearance differs from the training rig.
• —Action jitter. Visible shaking during execution, likely action-chunk boundary discontinuity. Temporal ensembling was not applied and would probably help.
• —Undertrained. Loss was still dropping at 20k steps, and 75 episodes is a small dataset. More data and more steps are the obvious next levers.

Other checkpoints

A wrist-camera-only variant trained on the same dataset also exists, evaluated at 80% (4/5) under the same protocol. It needs only one camera, which may make it easier to wire into a new setup. Not uploaded here, but open an issue on ggand0/vla-so101 if it would be useful.

License

Apache 2.0, inherited from lerobot/smolvla_base.