RahulR3174/smolvla-6dof-pick-place-20k

SmolVLA 20K — 6-DoF Reference Arm Pick-and-Place

A fine-tuned SmolVLA policy for visual pick-and-place manipulation using a simulated 6-DoF reference arm.

The policy was trained for:

Pick up the red cube and place it on the blue target.

Model

• —Base architecture: SmolVLA
• —Vision-language backbone: HuggingFaceTB/SmolVLM2-500M-Video-Instruct
• —Training steps: 20,000
• —Action chunk size: 50
• —Control frequency: 20 Hz
• —Cameras: 2
• —Robot state: 6-D
• —Robot action: 6-D
• —Framework: LeRobot

Inputs

The policy expects four inputs.

1. Top camera

observation.images.camera1

Shape: (3, 384, 384)

2. Wrist camera

observation.images.camera2

Shape: (3, 384, 384)

Both images should be RGB and represented as floating-point tensors in the range:

[0, 1]

3. Robot state

observation.state

Shape: (6,)

The six values are:

[joint_1, joint_2, joint_3, joint_4, joint_5, gripper]

4. Task instruction

A natural-language instruction is provided as:

task

Example:

Pick up the red cube and place it on the blue target.

Output

The policy produces a 6-D logical action:

[joint_1, joint_2, joint_3, joint_4, joint_5, gripper]

The policy is action-chunked with a configured chunk size of 50.

The action should be passed through the checkpoint's included postprocessing pipeline before being sent to the robot/simulator.

Basic usage

After downloading this repository, load the policy from the model directory:

from pathlib import Path

import torch

from lerobot.policies.smolvla.modeling_smolvla import SmolVLAPolicy

model_path = Path("./smolvla-6dof-pick-place-20k")

policy = SmolVLAPolicy.from_pretrained(model_path).eval()

device = "cuda" if torch.cuda.is_available() else (
    "mps" if torch.backends.mps.is_available() else "cpu"
)

policy.to(device)

policy.reset()

observation = {
    "observation.images.top": top_image,
    "observation.images.wrist": wrist_image,
    "observation.state": robot_state,
    "task": [task_instruction],
}

action = policy.select_action(observation)

print(action)

Use the policy_preprocessor.json and policy_postprocessor.json included in this repository as part of the normal LeRobot policy inference pipeline. The saved processor files contain the preprocessing, normalization, device, tokenization, and action postprocessing configuration used with this checkpoint.

Trained task and evaluation

The policy was trained on a two-camera 20 Hz pick-and-place dataset using a reference 6-DoF arm.

The trained visual setup uses:

• —red cube
• —blue target
• —two RGB cameras
• —6-D robot state
• —natural-language task instruction

The model was evaluated in the Genesis simulator with randomized cube and target positions within the training workspace.

Known limitation

The model shows strong performance on the trained red cube → blue target visual configuration and remains robust to several changes in wording of the task instruction.

In experiments without additional training, changing the physical object/target colors (for example, green/yellow or orange/purple) caused a substantial drop in success rate.

Therefore, the checkpoint should be treated as a task-specific research model rather than a color-invariant general-purpose manipulation policy.

Training configuration

The final checkpoint was trained with:

• —2 × NVIDIA Tesla T4
• —Per-GPU batch size: 4
• —Effective batch size: 8
• —20,000 training steps
• —20 Hz data/training rate
• —Two visual observations
• —50-action chunks

The repository contains the final inference checkpoint.

Training-state files used for resuming training are not included in this inference model package.

Included files

The model repository contains:

• —model.safetensors — trained model weights
• —config.json — policy/model configuration
• —policy_preprocessor.json — input preprocessing configuration
• —policy_preprocessor_step_5_normalizer_processor.safetensors — saved normalization state
• —policy_postprocessor.json — output postprocessing configuration
• —policy_postprocessor_step_0_unnormalizer_processor.safetensors — saved action unnormalization state
• —train_config.json — training configuration metadata
• —demo/smolvla_successful_pick_place_demo_combined.mp4 — synchronized top + wrist camera demonstration

Project code

The complete Genesis environment, data-collection code, training scripts, evaluation scripts, and deployment code are maintained separately in the project GitHub repository.

This Hugging Face repository is intended to contain the model and model-specific usage information, not the complete project source code.