Datasetpublic

Cocoyawn32/vlanext-kairos-robolab-eval

VLANeXt-Kairos · RoboLab Evaluation Rollouts Rollout videos and per-episode metrics from evaluating VLANeXt-Kairos (QWorld world-model VLA) checkpoints in the RoboLab / Isaac-Sim simulator on a DROID (Franka Panda, joint-position) manipulation task. This repository is a long-lived evaluation archive. Each top-level folder is one evaluated checkpoint; new checkpoints / tasks are added as sibling folders over time (see Naming convention below). What's inside… See the full description on the dataset page: https://huggingface.co/datasets/Cocoyawn32/vlanext-kairos-robolab-eval.

sourceHugging Facecc-by-4.0updated 2mo agoView on Hugging Face

0likes70downloads

Dataset Card

VLANeXt-Kairos · RoboLab Evaluation Rollouts

Rollout videos and per-episode metrics from evaluating VLANeXt-Kairos (QWorld world-model VLA) checkpoints in the RoboLab / Isaac-Sim simulator on a DROID (Franka Panda, joint-position) manipulation task.

This repository is a long-lived evaluation archive. Each top-level folder is one evaluated checkpoint; new checkpoints / tasks are added as sibling folders over time (see Naming convention below).

What's inside

Folder	Checkpoint	Task	Envs	Success
`baseline_70k/`	baseline `ds_checkpoint_70000` (all-GATED DiT)	BananaInBowl ±20 cm random init	4	0 / 4
`ttt_60k/`	TTT-1/4 `ds_checkpoint_60000` (every-4th layer = TTT)	BananaInBowl ±20 cm random init	4	0 / 4

Each checkpoint folder contains:

—videos/ — 8 MP4s: 4 parallel environments × {policy view *_envN.mp4 (1280×360), third-person *_envN_viewport.mp4 (864×480)}, 15 fps, ~50 s each.
—episode_results.jsonl — one line per episode: success, kinematic-smoothness metrics, and timing.
—README.md — checkpoint description + the metrics reported here.

Evaluation setup (common to all folders)

—Simulator: RoboLab on Isaac-Sim; robot = DROID Franka Panda, action = 7 joint positions + 1 gripper (absolute, joint-position control @ 15 Hz).
—Task: BananaInBowlUniform20 — "Pick up the banana and place it in the bowl", banana & bowl initial pose uniformly randomized ±20 cm in x/y. 750 steps (50 s) cap.
—Observation: two RGB cameras (over_shoulder_left, wrist_cam), each resized to 180×320 server-side, horizontally concatenated (exterior|wrist), then resized to the QWorld world-model input 512×256 and VAE-encoded. bf16, flash-attention, 10 diffusion steps.
—Rollout: open-loop horizon 8 (execute 8 predicted actions, then re-query), 4 parallel environments, 1 run → 4 episodes per checkpoint.

Metrics (in `episode_results.jsonl`)

field	meaning
`success`	task success (banana released inside bowl with gripper detached)
`ee_path_length`	end-effector path length (m); lower = more direct
`joint_rmse_mean`	mean per-joint RMSE vs. reference
`ee_sparc` / `joint_sparc_mean`	spectral-arc-length smoothness (less negative = smoother)
`ee_isj` / `joint_isj`	integrated squared jerk (lower = smoother)
`ee_speed_max` / `ee_speed_mean`	end-effector speed (m/s)
`timing.policy_inference_avg_ms`	avg server inference latency per query

Note on success = 0. RoboLab is a simulator; the checkpoints were trained on real-robot DROID data (MolmoAct2-DROID). There is an inherent sim-to-real domain gap, so low simulation success on the harder ±20 cm-randomized task is expected and does not indicate a broken pipeline. The videos + kinematic metrics are the primary artifact for qualitative/behavioral comparison.

Naming convention (for long-term maintenance)

—Repo: vlanext-kairos-robolab-eval — <project>-<simulator>-<purpose>.
—Per-checkpoint folder: <variant>_<step/1000>k (e.g. baseline_70k, ttt_60k).
—Adding a new run: create a new sibling folder; if the same checkpoint is re-evaluated on a different task, suffix the task, e.g. ttt_60k_<task>. Keep videos/, episode_results.jsonl, and a folder README.md in every folder.

Provenance