Datasetpublic

ISRHUMANOID/ur3-bimanual-lingbot-va

UR3 Bimanual Robot Dataset for LingBot-VA Fine-tuning 202 teleoperated episodes of a bimanual UR3 robot performing manipulation tasks, preprocessed and ready for LingBot-VA fine-tuning. Dataset Summary Property Value Episodes 202 Unique tasks 97 Total frames 61,074 (at 30 fps) Cameras 3 (top, left wrist, right wrist) Action space 30-dim (14 active: both arms EEF + grippers) Format LeRobot v2.1 + LingBot-VA pre-extracted latents… See the full description on the dataset page: https://huggingface.co/datasets/ISRHUMANOID/ur3-bimanual-lingbot-va.

sourceHugging Faceapache-2.0updated 4mo agoView on Hugging Face

1likes652downloads

Dataset Card

UR3 Bimanual Robot Dataset for LingBot-VA Fine-tuning

202 teleoperated episodes of a bimanual UR3 robot performing manipulation tasks, preprocessed and ready for LingBot-VA fine-tuning.

Dataset Summary

Property	Value
Episodes	202
Unique tasks	97
Total frames	61,074 (at 30 fps)
Cameras	3 (top, left wrist, right wrist)
Action space	30-dim (14 active: both arms EEF + grippers)
Format	LeRobot v2.1 + LingBot-VA pre-extracted latents

Task Coverage

Tasks span 6 verb categories across ~20 object classes, covering both single-arm and bimanual operations:

—Pick / Place — reaching, grasping, and depositing objects
—Push / Pull — non-prehensile contact manipulation
—Roll — sliding cylindrical objects
—Share — handing objects from one arm to the workspace
—Bimanual pass — pick with right arm, pass to left arm (108 episodes)

Objects include: balls (green, red, blue, yellow), cubes, cup, bottle, cylinder, bag, book, box, eraser, plant, sauce, socks, stapler, tape measure, white cup.

Robot Setup

—Arms: 2× Universal Robots UR3 (6-DoF each)
—End-effectors: Parallel jaw grippers
—Cameras:
—observation.images.cam_high — top/front overview camera (1280×720 raw, 256×320 in latents)
—observation.images.cam_left_wrist — left wrist camera
—observation.images.cam_right_wrist — right wrist camera
—Data collection: Human teleoperation at 30 fps

Action Space

Dims	Description
0–5	Left arm end-effector pose (x, y, z, roll, pitch, yaw)
6	Left arm gripper (binary: 5=closed, 220=open)
7–12	Right arm end-effector pose
13	Right arm gripper (binary: 5=closed, 190=open)
14–29	Unused (zeros) — reserved for future expansion

Active dims: 0–13 (used_action_channel_ids = list(range(14))).

Normalization: quantile (action_norm_method = 'quantiles') using q01/q99 stats stored in wa_ur3_cfg.py.

Directory Structure

ur3-bimanual-lingbot-va/
├── README.md
├── empty_emb.pt              # T5/UMT5-XXL embedding of "" — for CFG training
├── meta/
│   ├── info.json             # LeRobot v2.1 dataset metadata
│   ├── episodes.jsonl        # Per-episode metadata with action_config
│   ├── episodes_stats.jsonl  # Per-episode action statistics
│   ├── tasks.jsonl           # Task text registry
│   └── filter_report.txt     # Quality filtering log
├── data/
│   └── chunk-000/
│       └── episode_XXXXXX.parquet   # Action + state data (30-dim float32)
├── videos/
│   └── chunk-000/
│       ├── observation.images.cam_high/
│       ├── observation.images.cam_left_wrist/
│       └── observation.images.cam_right_wrist/
│           └── episode_XXXXXX.mp4  # Raw 1280×720 @ 30fps
└── latents/
    └── chunk-000/
        ├── observation.images.cam_high/
        ├── observation.images.cam_left_wrist/
        └── observation.images.cam_right_wrist/
            └── episode_XXXXXX_0_END.pth  # Pre-extracted VAE latents

Pre-extracted Latents Format

Each .pth file under latents/ contains a dict:

python

{
    'latent':            torch.Tensor,  # [N*H*W, C] bfloat16, H=16, W=20, C=4
    'latent_num_frames': int,           # N (number of 10fps latent frames)
    'latent_height':     16,            # spatial H after 16× VAE compression
    'latent_width':      20,            # spatial W after 16× VAE compression
    'video_num_frames':  int,           # frames in original 10fps video
    'video_height':      256,
    'video_width':       320,
    'text_emb':          torch.Tensor,  # [1, seq, 4096] bfloat16 — UMT5-XXL encoding
    'text':              str,           # task instruction string
    'frame_ids':         list[int],     # 10fps indices (0, 1, 2, ...) used
    'start_frame':       int,
    'end_frame':         int,
    'fps':               10,
    'ori_fps':           30.0,
}

Latents were extracted with Wan2.2 VAE (robbyant/lingbot-va-base) at 256×320 resolution, 10fps (every 3rd frame of 30fps raw data).

Usage — LingBot-VA Fine-tuning

Prerequisites

bash

git clone https://github.com/RobbyAnt/lingbot-va
cd lingbot-va
source activate.sh

# Download base model
huggingface-cli download robbyant/lingbot-va-base --local-dir models/lingbot-va-base

# Download this dataset
huggingface-cli download ISRHUMANOID/ur3-bimanual-lingbot-va \
    --repo-type dataset --local-dir data/ur3_combined

Single GPU (RTX 5090, 31 GB)

bash

# Edit transformer/config.json: set attn_mode = "flex" for training
NGPU=1 CONFIG_NAME='ur3_train' bash script/run_va_posttrain.sh

Multi-GPU / A100

bash

NGPU=1 CONFIG_NAME='ur3_train_a100' bash script/run_va_posttrain.sh

Key training config (wan_va/configs/va_ur3_train_cfg.py):

Parameter	RTX 5090	A100
`max_latent_frames`	8	16
`batch_size`	1	2
`gradient_accumulation_steps`	8	4
`lora_rank`	16	32
`num_steps`	10,000	15,000
`cfg_prob`	0.25	0.25
`subsegment_step_lat_frames`	4	2

Data Quality Notes

—3 episodes removed: too short (< 3 seconds after downsampling) — episodes 156, 159, 162 from the original 205
—Gripper spikes: Dims 6 and 13 (gripper) show large single-step deltas (up to 220 units) — this is expected binary open/close behavior, not data corruption
—Sub-segmentation: The dataset loader creates ~1,010 training items from 202 episodes by sliding a window with step=4 latent frames, preventing memorization

Citation

If you use this dataset, please cite the LingBot-VA paper and acknowledge the ISR Humanoid Lab dataset.