Voxel51/boreas-multimodal

Dataset Card for Boreas Multimodal (MCAP)

[image]

A FiftyOne build of Boreas and Boreas Road Trip (Boreas-RT), the multi-season and multi-route autonomous driving datasets from the Autonomous Space Robotics Laboratory (ASRL) at UTIAS. This build repackages 3 driving sequences and 6 object-detection windows as time-synchronized MCAP recordings for FiftyOne's native multimodal dataset support (FiftyOne 1.19+). Each sample is one episode, viewable in FiftyOne's tiled multimodal viewer with synchronized camera, 128-beam lidar point cloud, decoded 360° radar imagery, GPS/map track, and IMU/encoder/velocity telemetry — plus, for the object-detection episodes, 3D bounding boxes rendered natively in the 3D tile and as 2D wireframe overlays on the camera and radar images.

Boreas/Boreas-RT are raw multi-sensor datasets for benchmarking odometry/localization algorithms across seasons, weather, and road types; only one recording (boreas-objects-v1) carries 3D object annotations (Car/Pedestrian/Cyclist/Misc). This repackaging does not add or alter any ground truth; see Dataset Creation for exactly what was kept, converted, and left out.

This is a FiftyOne dataset with 8 samples.

Installation

If you haven't already, install FiftyOne:

pip install -U fiftyone

Usage

import fiftyone as fo
from fiftyone.utils.huggingface import load_from_hub

# Load the dataset
# Note: other available arguments include 'max_samples', etc
dataset = load_from_hub("Voxel51/boreas-multimodal")

# Launch the App
session = fo.launch_app(dataset)

Dataset Details

Dataset Description

Boreas is a multi-season autonomous driving dataset collected by repeatedly driving a route near the University of Toronto over one full year (Nov 2020-Nov 2021), capturing sun, rain, falling snow, and every season in between — 44 public sequences, 350 km total. Boreas Road Trip (Boreas-RT) extends this with 9 new, more diverse routes (suburbs, industrial, urban, forest, farm, tunnel, skyway, regional, freeway), each repeated on average 6 times — 60 sequences, 643 km total — adding an Aeva Aeries II FMCW Doppler lidar (32 of 60 sequences) and a standalone Silicon Sensing DMU41 IMU on top of the original sensor suite (128-beam Velodyne Alpha-Prime lidar, 360° Navtech scanning radar, 5MP FLIR Blackfly S camera, Applanix POS LV GNSS-INS). A separate, dedicated recording (boreas-objects-v1, ~104 minutes, sunny weather only) provides 326,180 3D object-detection annotations (Car/Pedestrian/Cyclist/Misc) across 7,111 labeled lidar frames. This FiftyOne build covers 2 driving-sequence episodes (one original Boreas, one Boreas-RT) and 6 object-detection windows carved from boreas-objects-v1 (see Curation Rationale).

• —Curated by: Autonomous Space Robotics Laboratory (ASRL), University of Toronto Institute for Aerospace Studies (UTIAS), in collaboration with Applanix/Trimble — original Boreas/Boreas-RT data collection, sensor calibration, and ground-truth pose post-processing. 3D object annotations for boreas-objects-v1 were produced by the Scale.ai data annotation service. This MCAP/FiftyOne multimodal repackaging (episode authoring, dataset card) was prepared independently by Harpreet Sahota.
• —Funded by: Natural Sciences and Engineering Research Council of Canada (NSERC); Applanix/Trimble (co-authorship and sensor support); General Motors (vehicle donation); hosting via the Amazon Open Data Sponsorship Program.
• —Shared by: Harpreet Sahota (this repackaging); the original Boreas and Boreas-RT datasets are shared by ASRL via the AWS Open Data Registry (s3://boreas) and https://www.boreas.utias.utoronto.ca.
• —Language(s): N/A (sensor data — camera, lidar, radar, GPS, IMU; no text).
• —License: Unknown — the AWS Open Data Registry listing's License field is blank, and no license is stated in the pyboreas devkit or bucket contents at the time of this card; see Citation for the registry's suggested access citation.

Dataset Sources

• —Repository (devkit): https://github.com/utiasASRL/pyboreas
• —Paper: Burnett, K., Yoon, D. J., Wu, Y., Li, A. Z., Zhang, H., Lu, S., Qian, J., Tseng, W.-K., Lambert, A., Leung, K. Y. K., Schoellig, A. P., & Barfoot, T. D. (2023). Boreas: A Multi-Season Autonomous Driving Dataset. The International Journal of Robotics Research, 42(1-2), 33-42. arXiv:2203.10168; Lisus, D., Papais, K. M., Le Gentil, C., Preston-Krebs, E., Lambert, A., Leung, K. Y. K., & Barfoot, T. D. (2026). Boreas Road Trip: A Multi-Sensor Autonomous Driving Dataset on Challenging Roads. arXiv:2602.16870
• —Demo: https://www.boreas.utias.utoronto.ca (official docs/download/ leaderboard site) and https://registry.opendata.aws/boreas/ (AWS Open Data listing)

Uses

Direct Use

• —Exercising/demoing FiftyOne's multimodal MCAP support: synchronized playback of camera, 128-beam lidar point cloud, decoded 360° radar imagery, GPS/map track, and IMU/encoder/velocity telemetry across real autonomous-driving recordings.
• —Inspecting 3D object-detection ground truth (Car/Pedestrian/Cyclist boxes) rendered natively as foxglove.SceneUpdate cuboids in the 3D tile, plus 2D wireframe overlays on the camera and radar bird's-eye-view images (both carrying label/uuid/numPoints metadata, inspectable via FiftyOne's built-in click-to-inspect panel), across 6 diverse time windows spread through the ~104-minute boreas-objects-v1 recording.
• —Prototyping odometry/localization analyses against dense ENU-frame ground-truth pose (/tf, /odom, /gps, /velocity) at each sensor's native timestamp.
• —Comparing sensor suites between the original Boreas platform and the Boreas-RT extension (adds an Aeva FMCW Doppler lidar and a standalone DMU41 IMU).

Out-of-Scope Use

• —Reproducing the official odometry/localization/detection leaderboards at boreas.utias.utoronto.ca — those benchmarks run over full, un-windowed sequences (44 Boreas + 60 Boreas-RT sequences, each 90-150+ GB raw); this build covers only 2 driving sequences (each a 10-second window, not the full sequence) plus 6 short windows carved from the single boreas-objects-v1 recording.
• —Training or evaluating on obj_test-split windows for detection — ground truth for that split is withheld by the source dataset for leaderboard fairness (confirmed directly: every obj_test window checked has zero label files); only obj_train windows are included here.
• —Any task requiring depth maps, semantic/instance segmentation, or an HD map layer — none exist anywhere in the source dataset (the devkit's own BoreasDataset.map attribute is an unimplemented TODO).

Dataset Structure

This is a flat (ungrouped) FiftyOne dataset with media_type: "multimodal" and 8 samples. Each sample is one episode, stored as one .mcap file; FiftyOne infers the multimodal media type automatically from the .mcap extension. There are no separate per-frame image or point-cloud samples — the episode is the sample unit, and every stream inside it (camera, lidar, radar, GPS, IMU, encoder, and — where present — 3D box annotations) is decoded live by FiftyOne's multimodal viewer.

The dataset carries no per-sample tags and dataset.info is empty (no extra dataset-level metadata beyond the per-sample fields below).

Episodes in this dataset

The 2 driving episodes are fixed 10-second windows starting mid-sequence (not full sequences — see Curation Rationale). The 6 boreas-objects-v1 episodes are 5 obj_train windows plus the devkit's own obj_sample window, spread across the full ~104-minute recording for scene diversity; ep000_objsample's window is also one of the obj_train windows. boreas-objects-v1 has no applanix/imu.csv/dmi.csv/lat-lon columns at all (an older, 2020-era recording — see Parsing decisions), hence has_gps is False for all 6.

Fields

Standard FiftyOne bookkeeping fields (id, metadata, created_at, last_modified_at) are also present but not source-specific.

MCAP topics (inside each episode)

Label types and why

No FiftyOne sample-level label fields (`Detections`, `Detections3D`, etc.) are attached. Because each sample is a multi-second-to-minute continuous recording rather than a single frame, there is no single fixed-length list a sample-level label field could hold. Instead, the 3D object annotations for the 6 boreas-objects-v1 episodes are embedded as additional MCAP topics inside the same timeline as the sensor data (/boxes, /camera/boxes, /radar/boxes — see MCAP topics), decoded live by FiftyOne's multimodal viewer alongside the camera/lidar/radar tiles, exactly like the sensor topics themselves. The has_boxes sample field exists so episodes can be filtered without opening every MCAP file first, e.g. dataset.match(F("has_boxes")).

Parsing decisions

• —One sample = one episode. Full Boreas/Boreas-RT sequences are 90-150+ GB raw each (44+60 = 104 sequences total), so only 2 driving episodes (one original Boreas, one Boreas-RT) were authored, each a fixed 10-second window starting mid-sequence rather than the full recording.
• —Object-detection episode selection. boreas-objects-v1 is one continuous ~104-minute recording; the source dataset's own detection benchmark carves it into 53 short (17.6-71.9s) named windows (obj_train: 37, obj_test: 16) rather than splitting by whole sequence. 6 windows were selected from obj_train (spread across the full recording for scene diversity) — obj_test windows were excluded because their ground truth is withheld by the source dataset for leaderboard fairness (confirmed directly: every obj_test window checked has zero labels_detection/*.txt files in range, while every obj_train window checked has labels).
• —Boxes with `numPoints == 0` are dropped, matching the source devkit's own object_detection.ipynb tutorial (BoundingBoxes.filter_empty()), which always removes boxes with no supporting lidar returns before display. Had no effect on the specific windows included here (0 of 4,896 boxes checked have numPoints == 0), but the authoring script is defensive against it for other windows.
• —3D boxes are logged at every lidar frame (10 Hz), not just directly labeled ones (5 Hz) — the source only labels every other lidar frame directly; boxes on the in-between frames are produced via the devkit's own SO(3) interpolation (BoundingBoxes.interpolate(), needs the asrl-pylgmath package), the same codepath the devkit itself uses for camera/radar frames (which never land on a directly-labeled timestamp). This is a deliberate scope choice beyond the devkit's own tutorial, which only ever shows directly-labeled lidar frames.
• —Box dimension order is `[length, width, height]` (verified directly in the source paper's text: "(dx,dy,dz) represent the bounding box dimensions (length, width, and height)") — the readme.txt shipped alongside the label files in the S3 bucket incorrectly states the order as height, width, length; this build follows the paper, not the bucket's readme.txt.
• —`gps_post_process.csv`'s `latitude`/`longitude`/`heading` are in radians, not degrees (like the sibling roll/pitch columns, used directly with no conversion elsewhere in the devkit) — converted to degrees for foxglove.LocationFix.latitude/longitude (which expect degrees), while heading is passed straight through unconverted (it's already radians, matching LocationFix.heading's expected units).
• —Angular velocity columns in every pose CSV are stored in `z,y,x` order, not x,y,z — an intentional, paper-documented convention ("the flipped ordering... for backward compatibility"), not a bug; reordered to x,y,z before logging.
• —Aeva point fields are `[x,y,z,radial_velocity,intensity,signal_quality, reflectivity,time,point_flags]` (9 named fields, point_flags occupying 2 of the 10 float32 slots as one float64) — this exact order is confirmed by the Boreas-RT paper's text, resolving an internal inconsistency in the devkit itself (a generic per-point-time helper assumes the last column is always "time", which is only true for the 6-field Velodyne format, not the 10-field Aeva format).
• —Radar Doppler/offset correction is applied only to the Boreas-RT episode, gated on the devkit's own radar_chirp_type_upgrade_time constant — applying it to the original (pre-upgrade) Boreas radar would use a bogus chirp-type value (that sensor generation has no real per- azimuth chirp data) and corrupt the image.
• —Sensors are not hardware-synchronized — real per-sensor timestamps are used as-is for /tf, which is correct/expected behavior per both source papers, not an alignment bug.

Dataset Creation

Curation Rationale

Full Boreas/Boreas-RT sequences are far too large for a lightweight FiftyOne showcase — 104 sequences total (44 Boreas + 60 Boreas-RT), each 90-150+ GB raw, plus a separate 152 GB standalone recording for object detection. Rather than attempting exhaustive coverage, this build selects: one original-Boreas driving episode and one Boreas-RT driving episode (each a 10-second window, chosen to demonstrate the respective sensor suites — the Boreas-RT one specifically for its Aeva FMCW lidar and dual IMU streams), plus 6 object-detection windows spread across the full boreas-objects-v1 recording for scene diversity, all drawn from the already-labeled obj_train split.

Source Data

Data Collection and Processing

Per ASRL's published papers and devkit documentation: Boreas was collected by driving a repeated route (the "Glen Shields" route) near the University of Toronto over one year (Nov 2020-Nov 2021), captured in every season and weather condition including falling snow — 44 public sequences, 350 km total. Sensor suite: a 128-beam Velodyne Alpha-Prime lidar (360°×40° FOV, ~2.2M points/s, 10 Hz), a 360° Navtech CIR304-H scanning radar (0.9° angular resolution, 0.0596 m/200 m range pre-upgrade or 0.0438 m/250 m range post-upgrade, 4 Hz), a 5MP FLIR Blackfly S camera (2448×2048, 81°×71° FOV, 10 Hz, images pre-rectified and pre-anonymized for faces/license plates), and an Applanix POS LV 220 GNSS-INS (200 Hz, 2-4 cm RTX accuracy) providing centimetre-level post-processed ground-truth pose in a fixed ENU frame anchored to the first pose of the first sequence. A dedicated standalone recording, boreas-objects-v1 (~104 minutes, sunny weather), was separately annotated by the Scale.ai data annotation service with 326,180 3D object boxes (Car/Pedestrian/Cyclist/Misc) across 7,111 labeled lidar frames at 5 Hz.

Boreas Road Trip (Boreas-RT) extends the same collection methodology to 9 new routes across suburban, industrial, urban, rural, and highway environments (60 sequences, 643 km total), adding an Aeva Aeries II FMCW Doppler lidar (120°×30° FOV, ~1.0M points/s, 32 of 60 sequences) and a standalone Silicon Sensing DMU41 IMU (200 Hz) alongside the original sensor suite, plus Doppler-capable radar firmware (per-azimuth "chirp type" metadata enabling Doppler velocity extraction and undistortion).

For this repackaging: raw per-frame sensor files, calibration, and ground-truth CSVs were downloaded from the public, unsigned s3://boreas bucket (targeted time-windowed slices, not full sequences), decoded using the official pyboreas devkit (with asrl-pylgmath/asrl-pysteam added to unblock its 3D box interpolation code path, and foxglove-sdk for MCAP authoring), cross-checked byte-for-byte against real downloaded files rather than trusted from documentation alone, and packed into one .mcap file per episode. No sensor data or ground truth was synthesized, relabeled, or altered beyond the conversions documented in Parsing decisions.

Who are the source data producers?

Autonomous Space Robotics Laboratory (ASRL), University of Toronto Institute for Aerospace Studies (UTIAS), in collaboration with Applanix/Trimble — original Boreas/Boreas-RT data collection, sensor calibration, and ground-truth pose post-processing.

Annotations

Annotation process

3D bounding boxes for boreas-objects-v1 were produced by the Scale.ai data annotation service, per the original Boreas paper: 7,111 lidar frames were labeled at 5 Hz (double the labeling rate is achievable via interpolation, since the lidar itself runs at 10 Hz — see Parsing decisions), yielding 326,180 unique 3D box annotations across 4 classes (Car, Pedestrian, Cyclist, Misc), divided into 53 continuous scenes (20-70s each) and split into 37 training scenes and 16 test scenes (test-scene labels withheld for the official benchmark). No other annotation exists anywhere in either dataset — odometry/localization ground truth is automated GNSS/IMU/wheel-encoder post-processing, not manual annotation.

Who are the annotators?

The Scale.ai data annotation service, for the boreas-objects-v1 3D box annotations only. All other "ground truth" in this dataset (pose, velocity) is produced automatically by Applanix's POSPac software suite from GNSS/IMU/wheel-encoder measurements plus an RTX correction subscription — no manual annotation is involved.

Personal and Sensitive Information

Camera images are pre-anonymized by the original dataset producers: faces and license plates are blurred using the "Understand.ai Anonymizer" tool before public release, per the original Boreas paper. This repackaging performs no additional processing, re-identification, or redaction beyond what ASRL already released publicly.

Citation

BibTeX:

@article{burnett_ijrr23,
  author  = {Keenan Burnett and David J Yoon and Yuchen Wu and Andrew Z Li
             and Haowei Zhang and Shichen Lu and Jingxing Qian and
             Wei-Kang Tseng and Andrew Lambert and Keith YK Leung and
             Angela P Schoellig and Timothy D Barfoot},
  title   = {Boreas: A multi-season autonomous driving dataset},
  journal = {The International Journal of Robotics Research},
  volume  = {42},
  number  = {1-2},
  pages   = {33-42},
  year    = {2023},
  doi     = {10.1177/02783649231160195}
}

@misc{lisus_brrt26,
  author        = {Daniil Lisus and Katya M. Papais and Cedric Le Gentil
                   and Elliot Preston-Krebs and Andrew Lambert and
                   Keith Y. K. Leung and Timothy D. Barfoot},
  title         = {Boreas Road Trip: A Multi-Sensor Autonomous Driving
                   Dataset on Challenging Roads},
  year          = {2026},
  eprint        = {2602.16870},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  url           = {https://arxiv.org/abs/2602.16870}
}

APA:

Burnett, K., Yoon, D. J., Wu, Y., Li, A. Z., Zhang, H., Lu, S., Qian, J., Tseng, W.-K., Lambert, A., Leung, K. Y. K., Schoellig, A. P., & Barfoot, T. D. (2023). Boreas: A multi-season autonomous driving dataset. The International Journal of Robotics Research, 42(1-2), 33-42.

Lisus, D., Papais, K. M., Le Gentil, C., Preston-Krebs, E., Lambert, A., Leung, K. Y. K., & Barfoot, T. D. (2026). Boreas Road Trip: A Multi-Sensor Autonomous Driving Dataset on Challenging Roads. arXiv:2602.16870.

"Boreas Autonomous Driving Dataset was accessed on DATE from https://registry.opendata.aws/boreas." — suggested citation text per the AWS Open Data Registry listing.

More Information

This repository is an independently-curated, derived subset of the official Boreas/Boreas-RT datasets, repackaged as MCAP for FiftyOne's multimodal support. It is not an official ASRL/UTIAS artifact. For the full datasets (all 104 driving sequences, the full boreas-objects-v1 recording, the devkit, and the live odometry/localization/detection leaderboards), see https://www.boreas.utias.utoronto.ca, https://registry.opendata.aws/boreas/, and https://github.com/utiasASRL/pyboreas.

Dataset Card Authors

Harpreet Sahota (@harpreetsahota) — MCAP repackaging and this card. Original dataset producers are listed under Dataset Description.

Dataset Card Contact

Harpreet Sahota — https://huggingface.co/harpreetsahota