wsagi/SmolVLA-PickOrange

SmolVLA-PickOrange

针对 LeIsaac SO-101 PickOrange 任务 LoRA-free 微调的 SmolVLA 策略 — 自训 15k step（main，sweep best）。 A fine-tuned [SmolVLA](https://huggingface.co/lerobot/smolvlabase) policy on the LeIsaac SO-101 PickOrange task. main = step-15000 (sweep best), full-parameter from lerobot/smolvla_base._

[image]

🔗 项目仓库 / Project repos：

• —vitorcen/isaaclab-experience — Isaac Lab + LeIsaac 多策略横评（parent project）— 含 7-baseline benchmark
• —vitorcen/LeIsaac-Training — LeIsaac fork（训练脚本 + 设计文档 / training scripts + design docs）

关于命名 / About the name：config.type=smolvla (LeRobot v1 SmolVLA implementation)，backbone 用 HuggingFaceTB/SmolVLM2-500M-Video-Instruct (SmolVLM2)。LeRobot 自己也叫 smolvla 而不是 smolvla2，所以仓库名沿用 SmolVLA-PickOrange。 `config.type=smolvla` (LeRobot v1 SmolVLA implementation) with `HuggingFaceTB/SmolVLM2-500M-Video-Instruct` backbone. LeRobot keeps the policy name `smolvla` (matching their naming), so this repo follows suit.

TL;DR

• —任务 / Task：Pick up the orange and place it on the plate — SO-101 单臂依次夹起 3 颗橙子并放盘子。
• —数据集 / Dataset：`LightwheelAI/leisaac-pick-orange` — 60 episode 遥操示范，30 fps，dual-cam 480×640。
• —架构 / Architecture：SmolVLA v1（450M），SmolVLM2-500M-Video-Instruct backbone + Action Expert，chunk_size=50。
• —训练 / Training：full-param 微调（无 LoRA），batch=8 / lr=1e-4 / 总 30k step 训练，30k 后明显过拟合。main = step-15000 (sweep best)。
• —评测 / Eval（Isaac Sim 5.1，5 round × 3 颗 = 15 颗，post-fix placement check）：
• —2/5 strict rounds, 8/15 oranges (53%), 133s avg ← 15k @ h=50
• —详见 `vitorcen/isaaclab-experience` README leaderboard

Checkpoint branches / ckpt 分支

Sweep 用 h=50 (= train chunksize), 5 round × 5 ckpt = 75 ep on Isaac Sim 5.1，单一 RTX 4090。_

Ckpt sweep 曲线 / Ckpt sweep curve

15k 是最佳点：训得久了开始 overfit 60 ep 这个小数据集，过早（10k 以下）尚未学到完整 pick-place-pick-place 长程序列。

oranges/15
  9 |
  8 |          ⭐ 15k
  7 |
  6 |       ●  20k
  5 |          ●  25k
  4 |             ●  30k
  3 |
  2 |
  1 |●  10k
  0 +----------------------
    10  15  20  25  30  k step

推理 inference 配置

# 1. 启 LeRobot async policy server
bash server/start_server.sh --lerobot-only

# 2. 跑 LeIsaac PickOrange eval
POLICY_CHECKPOINT=wsagi/SmolVLA-PickOrange \
ACTION_HORIZON=50 \
EVAL_ROUNDS=5 EPISODE_LENGTH=120 MAX_ROUND_WALL_S=180 \
PROMPT="Pick up the orange and put it in the plate" \
conda run -n isaaclab python LeIsaac/scripts/evaluation/policy_inference.py \
    --task=LeIsaac-SO101-PickOrange-v0 \
    --policy_type=lerobot-smolvla \
    --policy_port=8080 \
    --policy_checkpoint_path=$POLICY_CHECKPOINT \
    --policy_action_horizon=$ACTION_HORIZON \
    --eval_rounds=$EVAL_ROUNDS --episode_length_s=$EPISODE_LENGTH \
    --max_round_wall_s=$MAX_ROUND_WALL_S \
    --policy_language_instruction="$PROMPT" \
    --device=cuda --enable_cameras

关键 inference 参数 (per [scripts/benchmark/baselines_action_horizon.tsv](https://github.com/vitorcen/isaaclab-experience/blob/main/scripts/benchmark/baselines_action_horizon.tsv))：

• —action_horizon=50（= train chunk_size，h=40 实测略弱）
• —选 branch main 拿 best；或 ckpt-30k / 任何 ckpt-Nk 拿对应阶段。

训练配方

Training recipe

🚨 schema-free base 关键 fix：训练前必须用 `prepare_base.sh` 剥光 lerobot/smolvla_base 自带的 input_features / empty_cameras（默认 camera1/2/3 @ 256×256 会污染微调路径），否则训练时 schema 不对齐 → forward 报 KeyError 或 silent 训坏。

Eval 历史 / Eval history

License

Apache-2.0（继承自 lerobot/smolvla_base）。