faraway6/waymoqa-videomqa

WaymoQA — VideoQA test subset (mosaic frames)

This repository hosts the video portion of the WaymoQA test split, prepared for VideoQA evaluation. It contains the multi-view 3x3 mosaic frames for every Waymo scenario token that carries video questions in the test set.

Contents

What the data is

• —Source dataset: Waymo Open Dataset — End-to-End Driving (waymo_open_dataset_end_to_end_camera_v_1_0_0, test split).
• —QA annotations: the official WaymoQA dataset (Yu et al., arXiv:2511.20022), downloaded from the authors' release.
• —Scope: the video questions of the WaymoQA test split — 896 QA items over 54 unique scenario tokens.
• —Frames: 7,357 mosaic images ({token}_{frame_index:03d}.jpg).

Frame format

Each .jpg is a 3×3 mosaic combining the 8 synchronized E2E camera views of one timestep, with per-camera text labels, matching the official WaymoQA video-QA preprocessing:

[FRONT_LEFT] [FRONT] [FRONT_RIGHT]
[SIDE_LEFT]  (blank) [SIDE_RIGHT]
[REAR_LEFT]  [REAR]  [REAR_RIGHT]

Frames of a given token form a temporal sequence (the video). A token's clip is ~10 Hz.

Note on frame counts

Each video contains ~121–141 frames (the official per-token annotation lists ~147–150). The first few frames of each clip are not present in the public Waymo E2E test shards, so only the frames actually available in the source data are included here. This does not affect strided sampling used for evaluation.

How to use

Merge the split archive and extract:

cat mosaics.tar.part_* > mosaics.tar
tar -xf mosaics.tar -d mosaics

This yields a flat directory mosaics/ of JPEGs named {token}_{frame:03d}.jpg.

Select the video questions from the annotations:

import json
rows = [json.loads(l) for l in open("test.jsonl", encoding="utf-8")]
video_qa = [r for r in rows if r.get("type") == "video"]   # 896 items
# each item: {token, waymo_split, type, question, category, options[4], answer_index}

To build a video-QA input, group the mosaic frames by token (sorted by frame index) and pass them (sampled by a stride) together with the question / options.

Attribution & license

• —Waymo Open Dataset is released under the Waymo Dataset License Agreement for Non-Commercial Use. Raw sensor data is not redistributed here; only derived mosaic frames of a small subset are provided, following the original Waymo terms.
• —The QA annotations belong to the WaymoQA authors: Yu, Seungjun; Lee, Seonho; Kim, Namho; Shin, Jaeyo; Park, Junsung; Ryu, Wonjeong; Jung, Raehyuk; Shim, Hyunjung. WaymoQA: A Multi-View Visual Question Answering Dataset for Safety-Critical Reasoning in Autonomous Driving. arXiv:2511.20022.
• —Please cite the original WaymoQA paper and comply with the Waymo dataset terms.

Citation

@article{yu2025waymoqa,
  title={WaymoQA: A Multi-View Visual Question Answering Dataset for Safety-Critical Reasoning in Autonomous Driving},
  author={Yu, Seungjun and Lee, Seonho and Kim, Namho and Shin, Jaeyo and Park, Junsung and Ryu, Wonjeong and Jung, Raehyuk and Shim, Hyunjung},
  journal={arXiv preprint arXiv:2511.20022},
  year={2025}
}