YieumYoon/groot-bimanual-so100-crlbasket-004

GR00T N1.5 — bimanual SO-101, red cube to basket (projector-only)

Fine-tune of `nvidia/GR00T-N1.5-3B` for a language-conditioned bimanual pick-and-place task on a self-built 12-DoF SO-101 rig. CWRU senior capstone (CSDS 395), Fall 2025.

This is the adopted model of a two-way ablation on tuning scope — see Ablation below.

Task: "Grab the red cube and put it in a red basket"

Results on real hardware

Mean end-to-end inference latency: 211 ms (LeRobot latency benchmark). Physical execution latency is higher, bounded by the hardware.

Recorded rollouts are public — these are the raw evaluation episodes, not summary statistics:

• —`eval_…-004-all-pos-check` — 24 rollouts across cube positions
• —`eval_…-004-all-pos-obstacle` — 23 rollouts with an obstacle in the workspace
• —`eval_…-004-c-l2` / `c-r3` — 5 rollouts each at single positions

Training

• —Dataset: `bimanual-crlbasket-rblock-merged-00` — 360 episodes / 238k frames / 3 cameras
• —20,000 steps, batch size 12, bf16 via accelerate, single H200 — ~150 min
• —Tuning scope: tune_projector=true, tune_diffusion_model=false, tune_visual=false, tune_llm=false, lora_rank=0
• —Average loss 0.00657, largely converged by 10k steps, grad norm 0.346

Throughput note. Three video streams per sample make decoding, not GPU compute, the bottleneck. The conventional pairing (batch 256 / 16 workers) projected ~15 h; inverting it to batch 12 / 84 workers finished in ~2.5 h on the same GPU. Worth checking before renting a bigger card.

Ablation: tuning scope

Same dataset, same 20,000 steps, same batch size. The only variable is whether GR00T's diffusion action head is trained. Run twice — once at 180 episodes, once at 360.

Average loss differs by 0.00003 — no signal. In side-by-side rollouts the frozen-head variant reached the grasp more consistently, which is the first failure point in this task, so it was adopted and given the bulk of the evaluation.

This comparison was informal. Successes were counted by eye across a small number of rollouts, not scored against a fixed protocol, and the two variants were never run under a matched evaluation. Treat it as an observation that motivated a choice, not as a measured result.

What the run does support is narrower and firmer: the loss curves carried no information about which checkpoint to pick. Whatever separated these two models was invisible offline. If you are fine-tuning GR00T on a small single-task dataset, do not rank checkpoints by loss.

Untested hypothesis for the direction of the effect: with 360 episodes of one task, training the action head erodes the pretrained action prior, while freezing it and adapting only the projector preserves it. Note that the documented default in NVIDIA's and third-party SO-101 tuning guides is to train the action head, so this observation runs against the standard recipe and deserves a proper measurement before anyone relies on it.

Limitations

• —Single task, single scene, discretized object positions. Not expected to transfer to free placement.
• —~25% success is a demonstration of end-to-end capability, not a reliable manipulator. Comparable hardware deployments we surveyed would not be expected above ~50% without far more data.
• —Known failure mode: when the cube is occluded under an arm, the policy does not recover. More camera coverage is the obvious next step.
• —The tuning-scope ablation was not scored under a fixed protocol — see Ablation. The rollout recordings are published so anyone can score them independently.
• —robot_type reads bi_so100_follower — that is the LeRobot embodiment class. The hardware is SO-101; no preset supported this bimanual configuration, so a custom embodiment was registered.

Project context

Part of a five-person capstone that built a full pipeline: iOS LiDAR/photogrammetry scanning → Meta SAM 3D reconstruction and segmentation → a browser-based 3D goal-state editor → a VLM turning before/after scene views into a natural-language goal → this policy → SO-101 hardware. The rig, the dataset, the training setup and this model were my scope.

How to Get Started with the Model

For a complete walkthrough, see the training guide. Below is the short version on how to train and run inference/eval:

Train from scratch

lerobot-train \
  --dataset.repo_id=${HF_USER}/<dataset> \
  --policy.type=act \
  --output_dir=outputs/train/<desired_policy_repo_id> \
  --job_name=lerobot_training \
  --policy.device=cuda \
  --policy.repo_id=${HF_USER}/<desired_policy_repo_id>
  --wandb.enable=true

Writes checkpoints to `outputs/train/<desiredpolicyrepoid>/checkpoints/`._

Evaluate the policy/run inference

lerobot-record \
  --robot.type=so100_follower \
  --dataset.repo_id=<hf_user>/eval_<dataset> \
  --policy.path=<hf_user>/<desired_policy_repo_id> \
  --episodes=10

Prefix the dataset repo with eval\_ and supply --policy.path pointing to a local or hub checkpoint.

Model Details

• —License: apache-2.0