alibustami/miniVLA-Nav

MiniVLA-Nav v1

A Multi-Scene Simulation Dataset for Language-Conditioned Robot Navigation

<!-- > Ali Al-Bustami -->

Demo

<video controls preload="metadata" width="100%"> <source src="https://huggingface.co/datasets/alibustami/miniVLA-Nav/resolve/main/assets/montageallscenes.mp4" type="video/mp4"> <a href="https://huggingface.co/datasets/alibustami/miniVLA-Nav/blob/main/assets/montageallscenes.mp4">All-scenes montage</a> </video>

Nova Carter navigating to named objects across all four Isaac Sim environments.

Dataset Summary

MiniVLA-Nav v1 is a simulation dataset for the Language-Conditioned Object Approach (LCOA) task: given a short natural-language instruction, an NVIDIA Nova Carter differential-drive robot must navigate to the named object and stop within 1 m. Data were collected inside four photorealistic NVIDIA Isaac Sim 5.1 environments (Office, Hospital, Full Warehouse, Warehouse with Multiple Shelves).

Each of the 1,174 episodes pairs a language instruction with per-timestep, synchronized multimodal observations:

All sensors operate at 60 Hz (physics Δt = 1/60 s).

Supported Tasks

• —Language-Conditioned Object Approach (LCOA) — given a natural-language goal and front RGB-D observations, predict continuous (v, ω) or discrete 7×7 action tokens to drive a differential-drive robot within 1 m of the named object.
• —Behaviour Cloning / Imitation Learning — dense per-step expert labels enable direct supervised training.
• —OOD Generalisation — structured evaluation splits test template-paraphrase and object-category out-of-distribution robustness.

Multimodal Observations

Each timestep provides synchronized RGB, metric depth (float32, metres), and instance segmentation. The composites below show RGB (left) and depth colormap (right) from a mid-episode step.

Depth strip — consecutive frames from an office episode, showing depth (metres) as the robot approaches the target:

[image]

Scenes

Four photorealistic Isaac Sim environments, each with curated seen/held-out object categories:

Office

[image]

Hospital

[image]

Full Warehouse

[image]

Warehouse (Multiple Shelves)

[image]

Object Categories

12 categories total — 9 seen during training, 3 held out for OOD evaluation.

Seen categories:

Held-out (OOD): fire\extinguisher, whiteboard, barrel — appear only in `testood_obj` split.

Object Category Demo

<video controls preload="metadata" width="100%"> <source src="https://huggingface.co/datasets/alibustami/miniVLA-Nav/resolve/main/assets/montageofficecategories.mp4" type="video/mp4"> <a href="https://huggingface.co/datasets/alibustami/miniVLA-Nav/blob/main/assets/montageofficecategories.mp4">Office categories montage</a> </video>

All object categories navigated to in the Office scene.

Dataset Structure

v1/
├── dataset_meta.json            # Global metadata (scenes, camera, action space, splits)
├── assets/                      # README visual assets
├── splits/
│   ├── train_id.txt             # 261 episode IDs
│   ├── val_id.txt               #  41 episode IDs
│   ├── test_id.txt              #  50 episode IDs
│   ├── test_ood_obj.txt         #  37 episode IDs  (held-out object categories)
│   └── test_ood_lang.txt        #  36 episode IDs  (paraphrase OOD templates)
├── targets_office.yaml          # Per-scene object catalogs (3-D centroids)
├── targets_hospital.yaml
├── targets_full_warehouse.yaml
├── targets_warehouse_multiple_shelves.yaml
└── episodes/
    └── ep_{N:06d}/
        ├── meta.json                 # Full episode metadata
        ├── rgb_front/{t}.png         # 640×640 RGB frame at step t
        ├── depth_front/{t}.npy       # 640×640 float32 depth (m) at step t
        ├── seg_front/{t}.png         # 640×640 uint16 instance segmentation at step t
        ├── actions_continuous.npy    # (T, 2) float32 — (v_t, ω_t)
        ├── actions_tokens.npy        # (T, 2) int16  — discretized 7×7 tokens
        └── poses.npy                 # (T, 7) float32 — (x,y,z,qw,qx,qy,qz)

Episode Metadata (meta.json)

Each episode's sidecar JSON records the full configuration:

{
  "episode_id": "ep_000321",
  "scene_id": "full_warehouse.usd",
  "goal": {
    "target_category": "crate",
    "target_id": "crate_038",
    "goal_position_xyz_m": [-15.08, 10.77, 2.93]
  },
  "instruction": {
    "text": "Go to the crate.",
    "template_id": "train_01"
  },
  "spawn": { "tier": "mid", "spawn_to_target_dist_m": 3.574 },
  "rollout": {
    "num_steps": 219,
    "terminated_by": "success",
    "success": true,
    "collision_count": 0,
    "final_ne_m": 0.966,
    "trajectory_length_m": 2.61
  }
}

Splits

Language Instructions

Instructions are generated from slot-fill templates with {object} and {color} placeholders.

18 training templates (T1–T18), examples:

• —"Go to the {object}."
• —"Drive to the {object} and stop."
• —"Approach the {object}."
• —"Navigate to the {object}."
• —"Your destination is the {object}."

12 paraphrase-OOD templates (O1–O12), examples:

• —"Make your way to the {object}."
• —"Proceed to the {object}."
• —"Find the {object} and come to a stop."
• —"Close in on the {object}."

Note: Color-slot templates are suppressed in v1 — all targets carry color=unknown because USD assets do not expose material-color attributes through a standard prim API. Active pool: 13 train + 10 paraphrase-OOD templates.

Task Definition

LCOA formulation: Given instruction $\ell$ and observations $ot = (It^\text{RGB}, Dt)$, output actions $at = (vt, \omegat)$ such that the robot stops within $r_\text{success} = 1.0$ m of the target object centroid.

Action space:

• —Continuous: $(v, \omega) \in [0, 1]$ m/s × $[-1.5, 1.5]$ rad/s
• —Tokenized: each dimension quantized to 7 uniform bins → 49-token vocabulary

Episode termination:

• —Success — within 1 m and stationary for ≥ 5 consecutive steps
• —Collision — stall detected (no forward progress for ≥ 16 steps near obstacle)
• —Timeout — 1,000 steps reached without success

Only successful episodes are retained in the dataset.

Spawn Tiers

Trajectory diversity is ensured through three distance tiers:

Pearson correlation between spawn distance and trajectory length: r = 0.94.

Expert Controller

The data-collection expert is a proportional controller using pixel-level target visibility from the instance segmentation mask:

• —Target visible (≥ 32 px): angular correction from mask centroid column + depth-based speed
• —Target not visible: bearing-only proportional law from known goal position
• —Obstacle avoidance: speed clamped when depth in central foreground crop < 0.25 m

Rollout Statistics

NE = final navigation error (distance to goal at termination). TL = trajectory length.

Collection Setup

Loading the Dataset

import json
import numpy as np
from pathlib import Path
from PIL import Image

root = Path("v1")

# Load split
with open(root / "splits" / "train_id.txt") as f:
    train_ids = [line.strip() for line in f]

# Load an episode
ep_dir = root / "episodes" / train_ids[0]
meta = json.loads((ep_dir / "meta.json").read_text())

instruction = meta["instruction"]["text"]             # "Go to the monitor."
actions = np.load(ep_dir / "actions_continuous.npy")  # (T, 2) float32
tokens  = np.load(ep_dir / "actions_tokens.npy")      # (T, 2) int16
poses   = np.load(ep_dir / "poses.npy")               # (T, 7) float32

# Load frame t=0
rgb   = np.array(Image.open(ep_dir / "rgb_front" / "0.png"))   # (640, 640, 3)
depth = np.load(ep_dir / "depth_front" / "0.npy")              # (640, 640) metres
seg   = np.array(Image.open(ep_dir / "seg_front" / "0.png"))   # (640, 640) instance IDs

Citation

If you use MiniVLA-Nav v1 in your research, please cite:

@misc{albustami2026minivlanavv1multiscenesimulation,
  title={MiniVLA-Nav v1: A Multi-Scene Simulation Dataset for Language-Conditioned Robot Navigation}, 
  author={Ali Al-Bustami and Jaerock Kwon},
  year={2026},
  eprint={2605.00397},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2605.00397}, 
}

License

This dataset is released under the Creative Commons Attribution 4.0 International (CC BY 4.0) license.

Contact

Ali Al-Bustami - abustami@umich.edu