learner1119/ffw_sh5_n17_260820_left_h50_rel_vis

GR00T N1.7 · FFW-SH5 left arm · horizon 50 · relative · vision tower tuned

Fine-tune of nvidia/GR00T-N1.7-3B on learner1119/260820 (100 episodes / 44,136 frames, FFW-SH5, left arm only). One of four runs differing only in action representation and whether the vision tower is trained; this repo is `RELATIVE` · `tune_visual=True`.

Checkpoints are subfolders, each a complete weights-only checkpoint as the trainer wrote it (BF16, 2 shards, no optimizer state). Load one with model_path="learner1119/ffw_sh5_n17_260820_left_h50_rel_vis/checkpoint-50000" or download the subfolder.

Final run: 30h 59m on 4× A100 80GB at 2.23 s/step. Train loss only — no validation split (eval_strategy='no'); tune_visual=True reaching ~half the loss on 100 episodes may be overfitting. Compare on held-out rollouts before choosing.

Data and modality

The parquet stores 16-dim state/action (both arms) but only the left 8 move — over all 44,136 frames the right-arm dims have std ≤ 0.0022 rad and gripper_r is constant — so the modality config declares only left_arm (0:7) and left_gripper (7:8). The SH5 hand's open/close is interpolated by the teleop stack to a single 0–1 scalar, so it is treated as 1-dof. Camera: observation.images.agentview 480×640 as cam_head. Chunk: 50 steps (2.5 s at 20 fps).

left_arm targets are relative — deltas from the joint angles at the observation timestep, one reference for all 50 steps; the gripper stays absolute. Normalisation statistics are per-horizon-step (relative_stats.json).

Training

global batch 64 (16/GPU × 4, DeepSpeed ZeRO-2, bf16) · 50k steps · lr 1e-4 cosine, warmup 0.05, wd 1e-5 · statedropout 0.2 · tunellm False / tunevisual True / projector True / diffusion True · seed 42 · backbone nvidia/Cosmos-Reason2-2B (LLM layers ≤ 12). The modality config used is `ffwsh5left8h50relconfig.py at the repo root — import it to register NEWEMBODIMENT` before loading. `experimentcfg/` inside each checkpoint is verbatim.

Sibling runs: `ffw_sh5_n17_260820_left_h50_abs`, `ffw_sh5_n17_260820_left_h50_abs_vis`, `ffw_sh5_n17_260820_left_h50_rel`. Code: KimDoYoung1997/Isaac-GR00T @ n1.7-doyoung-a100.