mcity-ai/mcity-av-alpamayo

mcity-av-alpamayo

Mcity AV recordings in Alpamayo-1.5 / PhysicalAI-AV format

18 × 20 s driving clips recorded at the Mcity Test Facility (Ann Arbor, MI), converted into the layout used by nvidia/PhysicalAI-Autonomous-Vehicles, so they load with the `physical_ai_av` devkit.

Loading

import io, json, zipfile, pandas as pd
from physical_ai_av import egomotion, video

root = "."
cam = "camera_front_wide_120fov"
clip_id = pd.read_parquet(f"{root}/clip_index.parquet").index[0]

with zipfile.ZipFile(f"{root}/camera/{cam}/{cam}.chunk_0000.zip") as z:
    reader = video.SeekVideoReader(
        video_data=io.BytesIO(z.read(f"{clip_id}.{cam}.mp4")),
        timestamps=pd.read_parquet(
            io.BytesIO(z.read(f"{clip_id}.{cam}.timestamps.parquet")))["timestamp"].to_numpy(),
    )

with zipfile.ZipFile(f"{root}/labels/egomotion/egomotion.chunk_0000.zip") as z:
    df = pd.read_parquet(io.BytesIO(z.read(f"{clip_id}.egomotion.parquet")))
ego = egomotion.EgomotionState.from_egomotion_df(df).create_interpolator(
    df["timestamp"].to_numpy())

# An Alpamayo-1.5 sample at t = 1.6 s into the clip:
images, stamps = reader.decode_images_from_timestamps(
    1_600_000 - np.arange(4)[::-1] * 100_000)        # 4 frames, 0.4 s at 10 Hz
history = ego(1_600_000 - np.arange(16)[::-1] * 100_000).pose   # 16 waypoints at 10 Hz
future  = ego(1_600_000 + (np.arange(64) + 1) * 100_000).pose   # 64 waypoints, 6.4 s

Using with Alpamayo-1.5

physical_ai_av.PhysicalAIAVDatasetInterface hardcodes repo_id="nvidia/PhysicalAI-Autonomous-Vehicles", so alpamayo1_5.load_physical_aiavdataset cannot be pointed at this dataset. The layout is identical, so alpamayo_loader.py (shipped in this repo) reads it directly and returns the same dict — same keys, shapes, dtypes and frame conventions — as a drop-in replacement.

pip install alpamayo1.5 physical_ai_av        # brings in torch, einops, av
hf download mcity-ai/mcity-av-alpamayo --repo-type dataset --local-dir mcity-av-alpamayo

import torch, sys
sys.path.insert(0, "mcity-av-alpamayo")
from alpamayo_loader import load_mcity_clip
from alpamayo1_5.models.alpamayo1_5 import Alpamayo1_5
from alpamayo1_5 import helper

model = Alpamayo1_5.from_pretrained("nvidia/Alpamayo-1.5-10B", dtype=torch.bfloat16).to("cuda")
processor = helper.get_processor(model.tokenizer)

data = load_mcity_clip(root="mcity-av-alpamayo", t0_us=5_100_000)   # or root=None to stream

messages = helper.create_message(
    data["image_frames"].flatten(0, 1), camera_indices=data["camera_indices"])
inputs = processor.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=False,
    continue_final_message=True, return_dict=True, return_tensors="pt")

model_inputs = helper.to_device({
    "tokenized_data": inputs,
    "ego_history_xyz": data["ego_history_xyz"],
    "ego_history_rot": data["ego_history_rot"],
}, "cuda")

with torch.autocast("cuda", dtype=torch.bfloat16):
    pred_xyz, pred_rot, extra = model.sample_trajectories_from_data_with_vlm_rollout(
        data=model_inputs, top_p=0.98, temperature=0.6,
        num_traj_samples=1, max_generation_length=256, return_extra=True)

print(extra["cot"][0])                              # Chain-of-Causation reasoning
gt = data["ego_future_xyz"][0, 0, :, :2]            # ground-truth 6.4 s trajectory

load_mcity_clip returns image_frames (N_cam, 4, 3, 1080, 1920), camera_indices (N_cam,), ego_history_xyz (1,1,16,3), ego_history_rot (1,1,16,3,3), ego_future_xyz (1,1,64,3), ego_future_rot (1,1,64,3,3), plus timestamps — trajectories already rotated into the ego frame at t0, as the model expects. Clips are 20 s, so t0_us may range from ~1.6 s to ~13 s (it needs 1.6 s of history and 6.4 s of future).

One camera, not four. Alpamayo-1.5 defaults to four cameras (front-wide, front-tele, cross-left, cross-right) and this dataset has only front-wide, so camera_indices is [1]. The model accepts fewer cameras — see NVIDIA's inference_cam_num.ipynb — but accuracy degrades in scenarios that depend on the missing views.

Conventions

• —Timestamps — microseconds relative to each clip's start; negative values are normal (egomotion carries pre-clip history).
• —Anchor frame — origin at the rig position at clip t=0, yaw rotated to zero at t=0; pitch and roll stay gravity-referenced. An egomotion row exists at exactly t=0.
• —Rig frame — x forward, y left, z up. Camera frame — x right, y down, z forward.
• —Egomotion columns — timestamp, qx, qy, qz, qw, x, y, z, vx, vy, vz, ax, ay, az, curvature. Velocity/acceleration are Savitzky–Golay derivatives of the 100 Hz INS pose track; curvature is yaw rate over horizontal speed, zeroed below 0.1 m/s where it is undefined.

Provenance and caveats

conversion_manifest.json records every parameter. Read it before using the data:

• —The camera clock ran +36.98 s ahead of the INS in the raw bags; timestamps here are corrected for that measured bias (verified to 0-frame lag against INS yaw rate).
• —Frames are centre-cropped from a 1920×1200 sensor, not resized.
• —Intrinsics are self-calibrated from ego motion, not from a calibration target: f = 1238.3 px, HFOV 82.1°, cx = 977.6 px. The camera-slot name says 120fov because that is the key Alpamayo-1.5 expects; the recorded intrinsics are the measured ones.
• —Camera mount position (1.70, 0, 1.45 m) and vehicle dimensions are assumed defaults, not measurements. Replace them with the real rig calibration for any metric work.
• —Recorded at a closed test facility; no reasoning traces or obstacle labels are included.