espejelomar/so101-can-butler
SO-101 Can Butler Teleoperated demonstrations of a human-triggered can handover on a low-cost SO-101 arm: the robot stays still until a person's hand appears on the mat, then reaches, grasps a can, and hands it over. A SmolVLA fine-tune on this data — 20k steps, about $2.30 of rented RTX 4090 — grasped and delivered the can autonomously, verified 3 times out of 3 attempts. Model: espejelomar/smolvla-so101-can-butler. What it looks like A policy trained on… See the full description on the dataset page: https://huggingface.co/datasets/espejelomar/so101-can-butler.
SO-101 Can Butler
<img src="media/banner.png" alt="SO-101 arm grasping a can under an overhead depth camera" width="100%"/>
Teleoperated demonstrations of a human-triggered can handover on a low-cost SO-101 arm: the robot stays still until a person's hand appears on the mat, then reaches, grasps a can, and hands it over.
A SmolVLA fine-tune on this data — 20k steps, about $2.30 of rented RTX 4090 — grasped and delivered the can autonomously, verified 3 times out of 3 attempts. Model: `espejelomar/smolvla-so101-can-butler`.
<a class="flex" href="https://huggingface.co/spaces/lerobot/visualize_dataset?path=espejelomar/so101-can-butler"> <img class="block dark:hidden" src="https://huggingface.co/datasets/huggingface/badges/resolve/main/visualize-this-dataset-xl.svg"/> <img class="hidden dark:block" src="https://huggingface.co/datasets/huggingface/badges/resolve/main/visualize-this-dataset-xl-dark.svg"/> </a>
What it looks like
A policy trained on this data, grasping autonomously. Left: the overhead camera. Right: the wrist camera — what the robot itself sees as the jaws close.
<img src="media/grasp_dualcam.gif" alt="Dual-camera view of an autonomous grasp: overhead left, wrist right" width="100%"/>
Frames from the demonstrations themselves — the reach, and the moment of the grasp with the human hand already waiting on the mat:
<p float="left"> <img src="media/demoreach.jpg" width="49%"/> <img src="media/demograsp.jpg" width="49%"/> </p>
Dataset Description
- Robot: SO-101 follower arm (Feetech STS3215 servos), teleoperated from an SO-101 leader arm
- Episodes / frames: 44 / 14,324 at 15 fps
- Cameras: overhead Orbbec Gemini 2 (
observation.images.top) and a wrist camera (observation.images.wrist), both 640×480 - Recorded with: LeRobot v0.6.1,
lerobot_record - License: apache-2.0
The negatives are not filler. They are what teaches the policy that the arrival of a hand is the cue — a policy trained without them moves at an empty table.
Recommended exclusion: episodes 0 and 12. In both, the arm began moving before the hand entered frame, which corrupts exactly that trigger. All published results used the remaining 42.
Curation rationale: density over variety
This dataset replaces an earlier 47-episode version of the same task that never produced a working policy — 0 successful grasps in 22 real rollouts. The fix was not more data. It was less spread.
With roughly one demonstration per object position, the earlier set could not even fit its own training data: near-identical images mapped to different actions, and an L1-trained policy answers that ambiguity with the average — a pose that grasps nothing. Concentrating the same recording effort into one zone with one object removed the ambiguity.
If you are collecting demonstrations on a budget: go deep before you go wide.
Recording protocol
Per episode: place the can in the zone → episode starts with the arm at rest → the human hand enters low and to the right and stays → the operator teleoperates a single smooth reach, gripper opening wide during the approach → stop, settle, then squeeze fully closed → lift straight up → deliver → release.
Two details carry most of the value:
- The hand arrives during the episode. Its appearance is the trigger; a hand already in frame at the first observation teaches nothing.
- The squeeze is full. The operator closes the leader gripper to ~0.9 while the follower stalls at ~11.5, held open by the can. That stall is the signal below.
Automatic success verification
Grasps here can be scored with no vision and no human labels, using only the gripper's commanded versus achieved position during the hold.
These servos are position-controlled: the gripper reports the angle it was told to reach, so position alone cannot distinguish a grasp from closing on air. But a can physically prevents the motor from arriving:
score = median(achieved − commanded) over the hold
held if score ≥ 1.5The two populations separate with margin on both sides, and no false positive was observed. Every result quoted here and in the model card was scored this way from the recorded episode.
Data structure
Each frame carries observation.state and action as 6-vectors in LeRobot units — shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll in degrees and gripper in 0–100 — plus the two camera streams, timestamp, frame_index, episode_index, and task_index.
Note that action is the leader arm's pose (what the human commanded) while observation.state is the follower's achieved pose. The difference between them is the signal exploited above, and it is why the two are not interchangeable.
Limitations and scope
- One object (a foil-wrapped can, ~66 mm diameter) in one zone of the workspace. Generalization to other objects, lighting, and positions is untested.
- The task is human-triggered by design; this is not an autonomous picking dataset.
- The demonstrator is a single operator with a consistent style; hand appearance and approach are not diverse.
- 44 episodes is small. It is enough to fine-tune a pretrained VLA for this narrow task, not to train from scratch.
- Hardware quirk worth knowing if you extend this: the Gemini 2's depth stream cannot see the arm's orange PLA at all (zero valid depth returns on it), which rules out several obvious arm-detection tricks.
Citation
@misc{espejel2026so101canbutler,
title = {SO-101 Can Butler: human-triggered can handover demonstrations},
author = {Espejel, Omar},
year = {2026},
url = {https://huggingface.co/datasets/espejelomar/so101-can-butler}
}Built with LeRobot. Companion model: `espejelomar/smolvla-so101-can-butler`.
<details> <summary><b>Raw <code>meta/info.json</code></b></summary>
{
"codebase_version": "v3.0",
"fps": 15,
"features": {
"action": {
"dtype": "float32",
"names": [
"shoulder_pan.pos",
"shoulder_lift.pos",
"elbow_flex.pos",
"wrist_flex.pos",
"wrist_roll.pos",
"gripper.pos"
],
"shape": [
6
]
},
"observation.state": {
"dtype": "float32",
"names": [
"shoulder_pan.pos",
"shoulder_lift.pos",
"elbow_flex.pos",
"wrist_flex.pos",
"wrist_roll.pos",
"gripper.pos"
],
"shape": [
6
]
},
"observation.images.top": {
"dtype": "video",
"shape": [
480,
640,
3
],
"names": [
"height",
"width",
"channels"
],
"info": {
"is_depth_map": false,
"video.height": 480,
"video.width": 640,
"video.codec": "h264",
"video.pix_fmt": "yuv420p",
"video.fps": 15,
"video.channels": 3,
"has_audio": false,
"video.g": 2,
"video.crf": 30,
"video.preset": null,
"video.fast_decode": 0,
"video.video_backend": "pyav",
"video.extra_options": {}
}
},
"observation.images.wrist": {
"dtype": "video",
"shape": [
480,
640,
3
],
"names": [
"height",
"width",
"channels"
],
"info": {
"is_depth_map": false,
"video.height": 480,
"video.width": 640,
"video.codec": "h264",
"video.pix_fmt": "yuv420p",
"video.fps": 15,
"video.channels": 3,
"has_audio": false,
"video.g": 2,
"video.crf": 30,
"video.preset": null,
"video.fast_decode": 0,
"video.video_backend": "pyav",
"video.extra_options": {}
}
},
"timestamp": {
"dtype": "float32",
"shape": [
1
],
"names": null
},
"frame_index": {
"dtype": "int64",
"shape": [
1
],
"names": null
},
"episode_index": {
"dtype": "int64",
"shape": [
1
],
"names": null
},
"index": {
"dtype": "int64",
"shape": [
1
],
"names": null
},
"task_index": {
"dtype": "int64",
"shape": [
1
],
"names": null
}
},
"total_episodes": 44,
"total_frames": 14324,
"total_tasks": 2,
"chunks_size": 1000,
"data_files_size_in_mb": 100,
"video_files_size_in_mb": 200,
"data_path": "data/chunk-{chunk_index:03d}/file-{file_index:03d}.parquet",
"video_path": "videos/{video_key}/chunk-{chunk_index:03d}/file-{file_index:03d}.mp4",
"robot_type": "so_follower",
"splits": {
"train": "0:44"
}
}</details>