Voxel51/cmht-autonomous-driving

Dataset Card for CMHT Autonomous Driving Multimodal (MCAP)

[image]

This is a FiftyOne dataset with 4 samples.

Installation

If you haven't already, install FiftyOne:

pip install -U fiftyone

Usage

import fiftyone as fo
from fiftyone.utils.huggingface import load_from_hub

# Load the dataset
# Note: other available arguments include 'max_samples', etc
dataset = load_from_hub("Voxel51/cmht-autonomous-driving")

# Launch the App
session = fo.launch_app(dataset)

Dataset Details

Dataset Description

The CMHT Autonomous Dataset is a multi-sensor autonomous-driving dataset recorded around Hamilton, Ontario by the Centre for Mechatronics and Hybrid Technologies (CMHT) at McMaster University. A single vehicle rig combines a Velodyne HDL-32E LiDAR, a Retina-4fn mmWave radar, a Logitech Brio monocular RGB camera, a FLIR A65 thermal (IR) camera, and GPS/IMU (built into the LiDAR unit), recorded live with ROS2 (galactic) into ros2 bag recordings. The dataset consists of over 9,000 labeled frames captured at 10-20 Hz across four drives spanning dusk/clear and night/rain conditions in downtown Hamilton, with every LiDAR-detected object labeled with its 3D position, size, rotation, classification, and object ID.

This repository repackages the four raw ROS2 bag recordings as time-synchronized MCAP episodes for FiftyOne's native multimodal dataset support, with the original per-frame 3D tracklet labels (published separately by the authors as a "frame-by-frame extracted" release) embedded directly back into each episode's MCAP timeline as ROS2 vision_msgs detection topics, synced to the exact sensor message each label was originally annotated from. Each sample is one continuous drive, viewable in FiftyOne's tiled multimodal viewer with synchronized camera, thermal camera, LiDAR point cloud, radar, GPS, IMU, and 3D/2D object-detection playback.

• —Curated by: Howard Zhang, Ash Liu, Saied Habibi, Martin v. Mohrenschildt, and Ryan Ahmed (Centre for Mechatronics and Hybrid Technologies, McMaster University) — original data collection, sensor rig, and 3D tracklet labeling. This MCAP/FiftyOne multimodal repackaging (ROS2-bag-to-MCAP conversion, data-quality fixes, and re-embedding of the separately-published labels into the MCAP timeline) was prepared independently by Harpreet Sahota.
• —Funded by: None — per the paper's Acknowledgements: "This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors."
• —Shared by: Harpreet Sahota (this repackaging); the original CMHT Autonomous Dataset is shared by McMaster University via the Federated Research Data Repository (FRDR) and a MacDrive file share.
• —Language(s): N/A (sensor data — camera, thermal camera, LiDAR, radar, GPS/IMU; no text).
• —License: CC0 1.0, per the FRDR record. (The accompanying Data in Brief paper text itself is published under CC BY 4.0 — that license applies to the paper, not the dataset files.)

Dataset Sources

• —Repository: FRDR record (direct file host: MacDrive Seafile share)
• —Paper: Zhang, H., Liu, A., Habibi, S., Mohrenschildt, M. v., & Ahmed, R. (2025). CMHT autonomous dataset: A multi-sensor dataset including radar and IR for autonomous driving. Data in Brief, 60, 111552. https://doi.org/10.1016/j.dib.2025.111552
• —Demo: [More Information Needed]

Uses

Direct Use

• —Multi-sensor fusion research combining LiDAR, radar, monocular camera, and thermal (IR) camera, including comparing sensor efficacy across weather/lighting conditions (dusk/clear vs. night/rain).
• —Training or evaluating 3D and 2D object detection/tracking models against the embedded ground-truth Car/Truck/Van/Pedestrian/Bus/LongVehicle tracklets (object IDs are preserved for tracking-across-time use cases).
• —Exercising and demoing FiftyOne's multimodal MCAP support: synchronized playback of camera, thermal, 3D point cloud (LiDAR + radar), GPS/map, and embedded 2D/3D detection tiles across real autonomous-driving recordings.
• —Studying sensor-fusion performance in adverse conditions specifically, since 3 of the 4 drives were recorded at night in rain.

Out-of-Scope Use

• —Using this as a large-scale benchmark comparable to KITTI, nuScenes, or Waymo — this is 4 drives with roughly 9,000 labeled frames total, an order of magnitude smaller than those datasets (an explicitly stated limitation in the source paper).
• —Relying on LiDAR resolution comparable to newer 64+ channel sensors — the Velodyne HDL-32E is a 32-channel unit, lower resolution than many current autonomous-driving LiDARs (also a stated paper limitation).
• —Sensor-fusion use cases that require overlapping fields of view across all sensors everywhere in the scene — the rig's sensors only share overlapping FOV at the front of the vehicle (a stated paper limitation).
• —Pedestrian-heavy benchmarking or training — the paper notes the dataset has comparatively low pedestrian representation, which "may cause detrimental effects in AI training/testing."
• —Treating the radar/GPS/IMU streams in the 3 night_rain_* drives as frame-exact synchronized with the camera/LiDAR streams — see is_preliminary_sync in Parsing decisions.
• —Any use requiring the original raw radar stream for night_rain_2 — the radar-labeled frame release did not include this drive (see Parsing decisions).

Dataset Structure

This is a flat (ungrouped) FiftyOne dataset with media_type: "multimodal" and 4 samples. Each sample is one episode — a single continuous drive/recording session — stored as one .mcap file; FiftyOne infers the multimodal media type automatically from the .mcap file extension. There are no separate image, point-cloud, or video samples: the drive is the sample unit, and every stream inside it (camera, thermal camera, LiDAR, radar, GPS, IMU, and the embedded object-detection topics) is decoded live by FiftyOne's multimodal viewer. No dataset-level splits are provided by the source data; the 4 drives (dusk_clear_0, night_rain_0, night_rain_1, night_rain_2) are the natural grouping, captured in the drive_name field. The dataset itself carries no sample tags, and dataset.info is empty (no extra dataset-level metadata beyond the per-sample fields below; in particular, the camera/IR calibration matrices used to derive the 2D detections are not stored anywhere in the FiftyOne dataset — they were only used transiently at MCAP-conversion time).

Fields

Standard FiftyOne bookkeeping fields (id, tags, metadata, created_at, last_modified_at) are also present but not source-specific.

Label types and why

No FiftyOne label fields (`Detections`, `Detections3D`, etc.) are attached directly to the sample. Because each sample is a multi-minute continuous recording rather than a single frame, there is no single fixed-length list a sample-level label field could hold. Instead, the 3D and 2D object annotations are embedded as additional ROS2 message topics inside the same MCAP timeline as the sensor data, decoded live by FiftyOne's multimodal viewer alongside the camera/LiDAR/radar tiles, exactly like the sensor topics themselves:

• —`/Labels_3D` (vision_msgs/msg/Detection3DArray) — one message per labeled LiDAR frame. Each Detection3D carries the object's class (results[0].hypothesis.class_id, one of Car/Truck/Van/ Pedestrian/Bus/LongVehicle), a persistent tracking id (the source's obj_id), and a bbox (vision_msgs/msg/BoundingBox3D) giving the 3D position, size, and orientation in the LiDAR frame. The source label format's rotation (Euler roll/pitch/yaw in radians, only yaw non-zero in every sample observed) is converted to a quaternion (scipy.spatial.transform.Rotation, ZYX intrinsic order) because vision_msgs/msg/BoundingBox3D.center is a geometry_msgs/msg/Pose.
• —`/Labels_2D_front` and `/Labels_2D_ir` (vision_msgs/msg/Detection2DArray) — the same objects, projected onto the monocular and thermal camera image planes respectively, as normalized pixel bounding boxes (vision_msgs/msg/BoundingBox2D). These 2D boxes are derived, not part of the original annotation — the source dataset only ships 3D LiDAR-frame tracklets; the 2D boxes here were computed by this repackaging using the authors' own projection method (see Data Collection and Processing).

Both label topics are only published at timestamps that had a matching labeled frame in the source release — most LiDAR/camera frames in a drive are unlabeled, so /Labels_3D//Labels_2D_* messages are sparser than the sensor topics they're synced to (see Parsing decisions for coverage numbers). The has_3d_labels/has_2d_labels sample fields exist so episodes can be filtered without opening every MCAP file first, e.g. dataset.match(F("has_3d_labels") & F("is_preliminary_sync") == False).

Schemas present across episodes

All 4 episodes have the identical schema set: sensor_msgs/msg/Image (monocular + thermal camera, on separate topics), sensor_msgs/msg/Imu, sensor_msgs/msg/NavSatFix, sensor_msgs/msg/PointCloud2 (LiDAR + radar, on separate topics), vision_msgs/msg/Detection2DArray, and vision_msgs/msg/Detection3DArray. Topic names, however, are not consistent across drives (e.g. night_rain_0/1/2 use /PCL for LiDAR and /Cam_Image for the monocular camera, while dusk_clear_0 uses /Lidar and /Camera instead) — the has_* fields and this dataset's own conversion pipeline key off schema name, not topic string.

Parsing decisions

• —One sample = one episode. Each sample corresponds to a single raw ROS2 bag recording (one drive), never split into per-frame samples — FiftyOne's multimodal viewer handles playback and scrubbing within an episode.
• —GPS coordinates were converted from raw NMEA `ddmm.mmmm` to decimal degrees before being written into NavSatFix messages (e.g. raw 4315.387, -7951.7195 → 43.256, -79.862, correct for Hamilton, ON). Left unconverted, the Map tile would place every episode in the wrong location.
• —`header.stamp` was broken/unusable on the LiDAR and camera topics in every raw bag (a placeholder sec=0 with a non-wall-clock nanosec counter) and was replaced with the message's own bag log/receive time. Radar, IMU, and GPS headers already carried correct epoch timestamps and were left as-is.
• —`vision_msgs` message definitions were registered into the `rosbags` typestore (vision_msgs_types.py), copied from the ROS2 galactic vision_msgs package, since rosbags does not ship them by default and they are required to write Detection2DArray/Detection3DArray messages into the MCAP.
• —Label-to-sensor-message frame sync required content-based matching, not positional counting. The source dataset publishes per-frame labels against a separately-published, frame-by-frame "extracted" release (not the raw bag), and that release numbers each sensor's frames independently (e.g. lidar/00002000.pcd, label/00002000.json). An initial approach assumed "labeled frame N == the Nth bag message on that topic," which only holds for the shortest drive (night_rain_2) — on the other 3 drives, the raw bag contains extra per-topic messages that were dropped when the extracted release was built, and the drop count grows through the drive (e.g. on dusk_clear_0, labeled frame 2000's LiDAR scan is actually the 2137th /Lidar message in the bag). The fix, implemented in convert_bag_to_mcap.py, walks the bag once per sensor role and matches each labeled frame to its true bag message by exact content comparison (pixel equality for images; near-exact point-coordinate equality for LiDAR, since the raw bag stores float32 and the extracted PCDs store float64), using a small forward-sliding window to tolerate the rare local reordering also observed in the raw data (e.g. night_rain_2's IR camera has its first two frames swapped relative to bag arrival order). All 4 episodes were verified to have 0 unmatched labels across every sensor role on every drive after this fix.
• —`is_preliminary_sync` marks a real synchronization caveat, not a processing choice. Only the LiDAR triggers the monocular and IR cameras; radar, GPS, and IMU run independently and are only approximately time-aligned. Each night_rain_* drive's own source README.txt states this explicitly ("the lidar and cameras are synchronized but are missing timestamp information... radar and gps/imu data... is not synchronized with the lidar and cameras"), and only dusk_clear_0 ships without that warning and has near-perfectly matched per-modality frame counts across all sensors.
• —`night_rain_2` has no radar data in this dataset. Its frame-by-frame extracted release ships no radar/ folder at all (radar exists only in that drive's raw bag under the source's own sync caveat above), so its MCAP's /Radar topic reflects the raw bag's radar stream directly rather than a curated/labeled subset like the other 3 drives.
• —PCD double-precision (`x y z intensity`/`vr`, `SIZE 8`) files from the extracted release are used only as a content-matching fingerprint source for the frame-sync fix above, not published into the MCAP directly — the /Lidar//PCL and /Radar PointCloud2 topics in the MCAP are the raw bag's own (float32) point-cloud messages.
• —No ROS `/tf`/`/tf_static` topic exists in any source bag (every message header uses the placeholder frame_id: "map"), so sensor extrinsics are not resolvable via a transform tree — they only exist in the separate per-drive calibration JSON files, which were used at conversion time to compute /Labels_2D_front//Labels_2D_ir and are not themselves stored in the resulting FiftyOne dataset.

Dataset Creation

Curation Rationale

The source dataset's stated rationale (per the paper) is that a sensor fusion dataset combining radar and IR alongside the more common LiDAR and camera — across a range of weather/lighting conditions — was, to the authors' knowledge, first of its kind, and useful for developing and evaluating sensor fusion techniques that remain robust when individual sensors (LiDAR, camera) degrade in poor weather.

This FiftyOne repackaging's rationale is to make the full sensor + annotation stack explorable as a single, time-synchronized artifact using FiftyOne's multimodal MCAP support, rather than requiring users to separately parse raw ROS2 bags and cross-reference a disjoint, independently-numbered frame-extraction release to see labels alongside sensor data.

Source Data

Data Collection and Processing

Per the source paper: data was collected with a vehicle-mounted sensor platform consisting of a Velodyne HDL-32E LiDAR (10 Hz, 32 channels, 100 m ± 0.02 m range), a FLIR A65 thermal camera (30 Hz, 640×512 resolution), a Retina-4fn mmWave radar (20 Hz, 250 m range), and a Logitech Brio monocular camera (30/60 FPS, 13 MP), controlled by a central laptop running ROS2-galactic on Ubuntu Linux. The LiDAR triggers both cameras; radar and GPS/IMU run independently and are synchronized to the LiDAR by closest timestamp. Camera-LiDAR extrinsic/intrinsic calibration used a checkerboard method; IR calibration used a heated checkerboard with detachable black cells for contrast; radar-LiDAR alignment used a translation-only extrinsic (rotation was found negligible). Raw recordings were captured as ROS2 bags and separately post-processed into a frame-by-frame, per-sensor folder structure using ROS2's ApproximateTimeSynchronizer (50 ns tolerance).

For this repackaging, each drive's raw ROS2 bag (SQLite3 storage) was converted to a single .mcap file using the `rosbags` Python library, with the data-quality fixes and label re-embedding described in Parsing decisions. The 2D detection boxes were computed with a reimplementation of the projection math in the authors' own reference script (example/CMHT_projection.py): standard pinhole projection (p_camera = R @ p_lidar + t, then K @ p_camera and divide by depth), with R/t from the calibration file's 4×4 extrinsic and K from its 3×3 intrinsic, both row-major.

Who are the source data producers?

The Centre for Mechatronics and Hybrid Technologies (CMHT), Department of Mechanical Engineering, and the Department of Computing and Software, both at McMaster University, Hamilton, Ontario, Canada — vehicle-mounted sensor platform driven around downtown Hamilton.

Annotations

Annotation process

Per the source paper's Specifications Table, the extracted frames were manually labeled using SUSTechPoints. Every LiDAR-detected object was labeled with a 3D bounding box (position, size, rotation) and a classification in Car, Truck, Van, Pedestrian, Bus, LongVehicle, plus a persistent object ID for tracking across frames. The 2D detection boxes in this repository (/Labels_2D_front, /Labels_2D_ir) were not manually annotated — they are derived by projecting the same 3D labels onto the camera image planes, as described in Data Collection and Processing.

Who are the annotators?

[More Information Needed] — the paper documents the annotation tool (SUSTechPoints) and process but does not name individual annotators.

Personal and Sensitive Information

The paper's Ethics Statement states: "The authors have read the ethical requirements and confirm that this dataset does not involve any human subjects, animal experiments, or data from social media platforms." Nonetheless, this is real-world street driving footage recorded in downtown Hamilton, Ontario — the monocular and thermal camera streams may incidentally depict identifiable pedestrians, vehicles, and license plates, and the GPS stream records the vehicle's actual driving routes through public roads. No additional anonymization was applied by this repackaging.

Citation

BibTeX:

@article{zhang2025cmht,
  title={CMHT autonomous dataset: A multi-sensor dataset including radar and IR for autonomous driving},
  author={Zhang, Howard and Liu, Ash and Habibi, Saied and Mohrenschildt, Martin v. and Ahmed, Ryan},
  journal={Data in Brief},
  volume={60},
  pages={111552},
  year={2025},
  publisher={Elsevier},
  doi={10.1016/j.dib.2025.111552}
}

APA:

Zhang, H., Liu, A., Habibi, S., Mohrenschildt, M. v., & Ahmed, R. (2025). CMHT autonomous dataset: A multi-sensor dataset including radar and IR for autonomous driving. Data in Brief, 60, 111552. https://doi.org/10.1016/j.dib.2025.111552

More Information

This repository is an independently-curated MCAP/FiftyOne repackaging of the official CMHT Autonomous Dataset. It is not an official CMHT/McMaster artifact. For the original release (raw ROS2 bags, frame-by-frame extracted sensor data, and label JSON files as published by the authors), see the FRDR record and the Data in Brief paper.

Dataset Card Authors

Harpreet Sahota (@harpreetsahota) — MCAP repackaging, label re-embedding, and this card. Original dataset authors are listed under Dataset Description.

Dataset Card Contact

Harpreet Sahota — https://huggingface.co/harpreetsahota