Datasetpublic

mjf-su/PhysicalAI-DE-Evaluation

PhysicalAI-DE-Evaluation A held-out German evaluation set for the navigation planner: 19,999 records, each pairing a single front-camera frame with the corresponding past trajectory, future ground-truth waypoints, and a natural-language driving objective. Provenance Every record here was drawn from the pool of German scenes that were withheld from every training stage of the planner: the base VLA pretraining mix, the reasoning supervised fine-tuning (SFT) stage… See the full description on the dataset page: https://huggingface.co/datasets/mjf-su/PhysicalAI-DE-Evaluation.

sourceHugging Facecc-by-4.0updated 4mo agoView on Hugging Face

0likes15downloads

Dataset Card

PhysicalAI-DE-Evaluation

A held-out German evaluation set for the navigation planner: 19,999 records, each pairing a single front-camera frame with the corresponding past trajectory, future ground-truth waypoints, and a natural-language driving objective.

Provenance

Every record here was drawn from the pool of German scenes that were withheld from every training stage of the planner:

—the base VLA pretraining mix,
—the reasoning supervised fine-tuning (SFT) stage, and
—the GRPO reinforcement-learning stage.

In other words: no record in this directory was ever seen by the models that will be evaluated on it. This is the canonical "honest" DE eval set — use it to compare checkpoints without leakage concerns, and to measure cross-country generalisation against the sibling US set.

The set is DE-only by construction.

Directory layout

PhysicalAI-DE-Evaluation/
├── README.md                ← this file
├── dataset.jsonl            ← 19,999 records, one JSON object per line
├── build_manifest.json      ← reproducibility manifest (seed, counts, sources)
└── camera/
    ├── camera_00.zip        ← original packed frames (≤3 GB cap, ZIP_STORED)
    └── camera_01.zip

Image format

—Codec: PNG (cv2.IMWRITE_PNG_COMPRESSION=3)
—Resolution: 640 × 360
—Camera: camera_front_wide_120fov (front-facing, ~120° HFOV)
—Path layout: {chunk_name}/{scene_id}/{timestamp_us}.png, relative to this directory after unzipping the camera files.
—Frame selection: the frame whose timestamp is closest to clip_start_us + timestamp_us for the given scene.

Resolving a record's frame is therefore:

python

frame_path = root / record["chunk_name"] / record["scene_id"] / f'{record["timestamp_us"]}.png'

JSONL schema

Each line in dataset.jsonl is a single JSON object. Fields fall into two groups.

Identity / indexing (always present)

Field	Type	Meaning
`chunk_name`	str	Chunk directory the frame lives in (e.g. `"chunk_2798"`).
`scene_id`	str	UUID of the scene clip.
`timestamp_us`	int	Offset into the clip, in microseconds. Used to address the frame.

Trajectory + task (always present)

Field	Type	Meaning
`egomotion`	list[[x, y, θ]]	Past trajectory, ego-frame, 0.25 s spacing, ending at the present anchor `[0, 0, 0]`. Length 9 (= 2 s of history + anchor).
`ground_truth_waypoints`	list[[x, y, θ]]	Future trajectory, ego-frame, 0.25 s spacing. Length 24 (= 6 s of lookahead).
`objective`	str	Natural-language driving intent for the next few seconds (e.g. `"Drive straight along the highway"`).
`difficulty`	`"easy"` \	`"hard"`	See Difficulty split below.

Coordinates are right-handed ego-frame, units meters / radians, with +x forward and +y left.

The difficulty label is the only thing that distinguishes a hard row from an easy one in the schema — see Difficulty split below for what that label actually means. DE rows also omit the shard_id identity field that US rows carry, because every DE row sourced from the same shard.

Difficulty split

Difficulty	Count	Meaning
`hard`	2,860	Scenes whose base-model ADE exceeds the 90th-percentile threshold calibrated on US data. Use these when you want a quality bar that exercises the harder tail of the distribution.
`easy`	17,139	Uniformly random scenes from the held-out DE pool, drawn from the complement of the hard stratum. A broad coverage sample.

Why is 14.3% of DE labelled hard, rather than the 10% the threshold name suggests? The threshold is the base-model US p90 — applying it to DE flags every record where the model struggles at least as much as the US tail does. Because the base model's ADE distribution on Germany is slightly rightward-skewed relative to its US distribution, more than 10% of DE records cross the US-calibrated bar. The 14.3% hard fraction is itself a coarse-grained measure of that cross-country degradation.

The --hard flag in the evaluation harness keeps only the 2,860 hard rows; without it you evaluate on the full 19,999.

Using this dataset

This directory is the default data source for the DE run of the ADE evaluation harness:

bash

python GRPO-alignment/evaluation/AverageDisplacementError.py \
    --country DE \
    --models <model-subdir-on-HF>

The harness reads dataset.jsonl, resolves each frame via the path layout above, and reports an Average Displacement Error leaderboard. Add --hard to restrict the evaluation to the 2,860 hard rows.

For full provenance details (random seed, source paths, stratification rule) see build_manifest.json.