Modelpublic

jsiburian/vla0-3b-ur5e-cluttered-pick-3obj-120

sourceHugging Faceapache-2.0updated 5mo agoView on Hugging Face

0likes3downloads

Model Card

VLA-0 (3B) - UR5e cluttered pick-3obj-120

Vision-Language-Action policy fine-tuned from `Qwen/Qwen2.5-VL-3B-Instruct` on `jsiburian/ur5e-cluttered-pick-3obj-120-lerobot-v2` (120 teleoperated UR5e demonstrations spanning 12 cluttered-scene pick tasks across 3 distinct target objects, 30,409 frames @ 17 fps) using the VLA-0 recipe.

This is the cluttered-scene companion to `jsiburian/vla0-3b-ur5e-pick-3obj-90`: identical robot, camera setup, and action layout, but with distractor objects in the workspace and 1.3 x more episodes (120 vs 90) and 2.3 x more frames (30,409 vs 13,426).

Model details


Base model	`Qwen/Qwen2.5-VL-3B-Instruct`
Method	VLA-0 (full fine-tune, 1000 discrete action bins, paired with language instruction)
Robot	UR5e (7-DoF)
Cameras	extra (3rd-person) + wrist (tiled side-by-side)
Action horizon	8
History	1
Image size	224 x 224 (tiled), random crop 0.9, brightness/contrast/saturation/hue aug

Training


Hardware	4 x NVIDIA H200 (GPUs 0-3)
Per-GPU batch	16 (effective batch 64)
Optimizer	AdamW, effective lr=4e-5 (config lr 1e-5 x num_gpus), weight decay=1e-10
Schedule	Constant, AMP (bf16), FlashAttention-2
Iterations	15,840 (~8 outer epochs) - stopped early after the training loss flattened around 0.058 (the configured ceiling was 20,000 iters / ~10-11 outer epochs)
Final train loss (per-50-iter wandb point at iter 15,800)	~0.0575
Action mask aug prob	0.4
Wall-clock	9.05 hours

Loss is the standard VLA-0 next-token cross-entropy over discretized action tokens (vocab of 1000 bins per dim, 56 action tokens per sample). Random-guess baseline is ln(1000) ~ 6.91. A loss of ~0.058 corresponds to ~94% probability mass on the correct bin per token.

The W&B run with full loss curves and per-iter system metrics is at wandb.ai/jsiburian-the-university-of-tokyo/vla0/runs/t45t1e7d.

Files

Path	Purpose
`model-0000{1,2}-of-00002.safetensors`	Sharded Qwen2.5-VL weights
`config.json`, `generation_config.json`	HF model config
`preprocessor_config.json`, `chat_template.json`	Image + text processors
`tokenizer*.json`, `vocab.json`, `merges.txt`, `added_tokens.json`, `special_tokens_map.json`	Tokenizer
`dataset_stats.pkl`	Action de-normalization stats - required at inference
`training_config.yaml`	Full training-config snapshot from the run

dataset_stats.pkl is a Python pickle of {"out_ori_act": {"min": np.ndarray(7,), "max": np.ndarray(7,), "count": np.ndarray(1,)}}, which carries the per-dimension action min / max used to normalize / decode the 1000-bin action tokens.

Usage

Use the VLA-0 repo (or our RoboVerse-flavored fork) for inference; load weights directly with Qwen2_5_VLForConditionalGeneration.from_pretrained(...) and pair with dataset_stats.pkl for bin <-> continuous action conversion.

The dataset's task strings (12 of them) are short imperatives like "pick up the cube and place it on the plate", "grasp the carrot and drop it onto the plate", etc. Match the phrasing of your task prompt to one of those instructions when sampling actions.

Caveats

—Single-task imitation: no LR schedule, no test split, no held-out object. The model is fit hard to the 120-episode demonstration distribution.
—Two-camera tiled input - if you only have a third-person view at inference, results will be worse than what training metrics suggest.
—Stopped early (iter 15,840 / 20,000) once loss plateaued. The recipe was the same as the 90-episode run (vla0_ur5e_pick_3obj_90.yaml) with only the dataset and num_iters changed.