anthonytec2/OctoSense

<p align="center"> <img src="assets/banner_inline.png" alt="OctoSense" width="760"/> </p>

<p align="center"> <a href="https://arxiv.org/abs/2606.27317"><img src="https://img.shields.io/badge/arXiv-2606.27317-b31b1b?logo=arxiv" alt="arXiv"/></a> <a href="https://github.com/anthonytec2/OctoSense"><img src="https://img.shields.io/badge/Code-GitHub-181717?logo=github" alt="Code on GitHub"/></a> <a href="https://colab.research.google.com/drive/11hBm5dH--7HueAFCKtaAFnDAJul-52fb?usp=sharing"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open the quickstart in Colab"/> </a> </p>

New to OctoSense? A getting-started Colab notebook walks through loading a sequence and using each modality. Click Open in Colab above to run it in your browser, no setup required.

<p align="center"> <video autoplay loop muted playsinline width="100%" poster="https://huggingface.co/datasets/anthonytec2/OctoSense/resolve/main/assets/hero.gif"> <source src="https://huggingface.co/datasets/anthonytec2/OctoSense/resolve/main/assets/hero.mp4" type="video/mp4"/> <img src="assets/hero.gif" alt="OctoSense, synchronized multi-sensor driving across day, night, and degraded conditions" width="100%"/> </video> </p>

OctoSense is a time-synchronized, calibrated, multi-sensor dataset spanning multiple platforms, all sharing the same sensor rig. The bulk of the data is large-scale driving dataset: 371 sequences · 59 hrs · 2,474 km · 8.43 TB of urban, suburban, and rural driving (highway / residential / city) on Long Island and in Philadelphia, across sunrise, daytime, sunset, and nighttime, complemented by additional sequences captured on a boat and a Unitree quadruped. Every sequence carries synchronized stereo RGB, stereo event cameras, infrared, Ouster LiDAR, and IMU; the driving sequences additionally provide GPS, CAN, ground truth (depth, ego-motion optical flow, semantic segmentation), and a fused GPS + LiDAR-inertial odometry trajectory. Each sequence's recording `platform` (car / boat / unitree) is a column in metadata.parquet: see Additional platforms for the boat/unitree specifics.

Canonical key for every sequence is its recording datetime, rosbag2_YYYY_MM_DD-HH_MM_SS. Per-sequence attributes live in `metadata.parquet` (the viewer's table: a sensor-montage thumbnail, a GPS-track map, and all metadata columns per sequence).

Additional platforms — Boat & Unitree

<p align="center"> <video autoplay loop muted playsinline width="100%" poster="https://huggingface.co/datasets/anthonytec2/OctoSense/resolve/main/assets/platformsposter.png"> <source src="https://huggingface.co/datasets/anthonytec2/OctoSense/resolve/main/assets/platforms.mp4" type="video/mp4"/> <img src="assets/platformsposter.png" alt="Boat and Unitree-quadruple sensor data (synchronized RGB, event, infrared, LiDAR)" width="100%"/> </video> </p>

Alongside the 371 car/driving sequences, OctoSense includes sequences from two other platforms recorded with the same sensor rig, distinguished by the `platform` column in metadata.parquet (car / boat / unitree). No perception ground truth is provided for these platforms.

• —`boat/sess1/`: 9 sequences: No LiDAR odometry or fused trajectory, and the LiDAR is raw rather than motion-compensated: data.h5 stores the raw Ouster range/signal/ir/reflectivity images instead of the deskewed ouster/range_pcl point cloud. Note the boat data used a different version OS1-64 LiDAR.
• —`unitree/sess1/`: 2 sequences (Unitree Go2-W quadruped). Runs LiDAR-inertial odometry like the driving data (ouster/range_pcl + ouster/odom/* + hf_odom), and adds per-joint robot state under `/robot`: low_state (motor q/dq/tau/temperature (N,20) + foot force + body IMU + battery), sport (body pose/velocity/gait), low_cmd (motor commands); see Unitree ROS Msg Data. No GPS lock; no fused trajectory.

Sensors

Coordinate frames & calibration

[image]

All extrinsics are stored per-sequence under /calib in the sequence h5, named A_T_B: a 4×4 matrix that maps a point from frame B into frame A (e.g. imgl_T_ouster takes LiDAR points into the left-RGB frame). Frame abbreviations:

Provided extrinsics: imgl_T_imgr, imgl_T_imu, imgl_T_ir, imgr_T_imu, ir_T_imgl, the event-pair set (evl_T_evr, evl_T_imgl, …), ouster/imgl_T_ouster / ouster/ir_T_ouster, and calib/lidar_T_lidarimu (LiDAR IMU → LiDAR/sensor frame, from the Ouster factory imu_to_sensor_transform). Each sequence records which calibration it used via rgb_cal_id / imu_cal_id / lidar_cal_id (in the metadata). RKO-LIO odometry (ouster/odom/*) tracks the LiDAR pose in the Map frame; the fused_traj trajectory tracks the same LiDAR pose in the UTM-relative world frame (see Ground truth). The depth GT is rendered in the rectified left-RGB camera image (OpenCV stereoRectify, rect_T_raw applied).

Per-session calibration is released alongside the data, in Kalibr format: - Cameras (calibration-camchain.yaml), a 4-camera Kalibr chain: cam0 = RGB right (/flir_cam_right), cam1 = RGB left (/flir_cam_left), cam2 = event right (/event_camera_right, 640×480), cam3 = event left (/event_camera_left, 640×480). Each entry has pinhole intrinsics [fx,fy,cx,cy] + radtan distortion_coeffs; each cam after cam0 carries T_cn_cnm1 (transform from the previous cam in the chain). - Camera↔IMU (calibration-camchain-imucam.yaml, adds T_img[l/r]_imu) + the IMU noise model (calibration-imu.yaml). - LiDAR↔camera (lidar_calibration_results.yaml) and IR intrinsics + extrinsics (ir_calib_result.json). The processed h5 renames cam0/cam1 to the right/left RGB streams above (and cam2/cam3 are the right/left event cameras).

<p align="center"> <video autoplay loop muted playsinline width="100%" poster="https://huggingface.co/datasets/anthonytec2/OctoSense/resolve/main/assets/gtgridposter.png"> <source src="https://huggingface.co/datasets/anthonytec2/OctoSense/resolve/main/assets/gtgrid.mp4" type="video/mp4"/> <img src="assets/gtgrid_poster.png" alt="Ground-truth modalities: optical flow, semantic segmentation, metric depth, and local trajectory" width="100%"/> </video> </p>

Ground truth

• —Odometry: **RKO-LIO** LiDAR-inertial odometry giving the SE(3) LiDAR pose in the Map frame (the LiDAR first frame) with linear/angular velocity. The high-rate hf_odom/* is obtained by integrating the IMU between LiDAR keyframes.
• —Depth: sparse metric depth in the rectified left-RGB image (depth_cm, uint16 cm, 0 = invalid). Built by accumulating 61 LiDAR scans (≈6 s), removing dynamic objects (YOLO26-medium on the nearest RGB frame), keeping the minimum depth per pixel, and then projecting into the camera image.
• —Optical flow: derived, not stored: the ego-motion-induced flow (accumulated points projected into a future RGB frame). Regenerate from depth + poses with derive_flow.py.
• —Semantic segmentation: 19-class Cityscapes pseudo-labels (**EoMT**), on the 303 daytime sequences (has_seg = true); nighttime sequences have no labels.
• —Fused reference trajectory: RKO-LIO odometry + GPS fused in a pose graph, under `fused_traj` (T_world_lidar (N,4,4) + t) in a UTM-relative world frame anchored at the first GPS fix. Geo-referencing + fusion quality live as fused_traj group attrs (epsg/utm_zone/hemisphere, origin_{easting,northing,alt}_m, lever_xyz_lidar_m, GPS residual RMS + p90, confidence_tier). fused_traj/t is non-uniformly sampled: poses are emitted at the union of the LiDAR-odom (≈10 Hz) and GPS (≈5 Hz) timestamps (≈13 Hz combined), so inter-pose intervals vary (≈0.02–0.1 s).

Per-sequence files (<platform>/<session>/<bag_id>/)

Units/conventions: depth in cm (0=invalid); event timestamps in µs; all other h5 time arrays in seconds.

Experimental, `car/radar` (Mazda forward radar). Six object tracks decoded from the CAN data with the opendbc Mazda radar DBC (CAN IDs 865–870 = RADAR_TRACK_361..366). Layout: car/radar/track_{1..6}/ with t (M,) float64 main-clock seconds, dist (M,) uint16, ang (M,) int16, vrel (M,) int16. Values are stored raw (no scale/offset): dist=DIST_OBJ (sentinel 4095 = no track), ang=ANG_OBJ (signed 12-bit), vrel=RELV_OBJ (signed 11-bit).

Data format

All */t arrays are seconds on a common PPS-synced main clock (Kalman-smoothed); event times are µs on that same clock (t/1e6 → s). Align streams by timestamp: RGB runs ≈100 Hz vs LiDAR ≈10 Hz. Camera frames are raw (un-rectified): rectify with the per-camera intrinsics + radtan dist_coeffs (stored in the h5 and the camchain); the depth GT is already in the rectified left-RGB frame.

Main bag h5 (`data.h5`):

Events (`events.h5`): flat asynchronous event streams per side, plus the event-camera calibration:

Depth GT (`rgb_left_rect_depth.h5`): per-LiDAR-scan sparse depth in the rectified left-RGB image. N = n_lidar_frames; frame k ↔ LiDAR scan k. Carries its own timestamps (identical to data.h5 ouster/t) and index arrays:

Root attrs: depth_scale_cm, flow_{gap,scale,invalid}, flow_stored (=False: flow is derived, not stored), n_scans.

Semantic GT (`rgb_left_rect_semantic.h5`, day sequences only): per-LiDAR-scan 19-class seg in the rectified left-RGB image, aligned 1:1 with rgb_left_rect_depth.h5:

Root attrs: classes (19 names), num_classes=19, model=EoMT-Cityscapes-DINOv2-L-1024, coordinate_frame=rectified_left, preprocessing=rectify+CLAHE, resolution=1920x1456. So semantic[k] ↔ depth_cm[k] ↔ img_left.mp4[left_img_indices[k]] ↔ ouster/t[k].

Captions & semantic search (`captions.h5`): each sequence is split into ≈5 s windows (W per sequence ≈ duration / 5 s); every window gets a natural-language scene caption (a Gemma-4-31B VLM) and a 4096-d text embedding (Qwen3-Embedding-8B) for text retrieval:

Root attrs: caption_model, embed_model, embed_dim (4096), num_windows, video_id.

Splits

train = 293 · test = 78. The dataset contains daytime, nighttime, and degraded sequences.

metadata.parquet fields (28)

platform ∈ {car, boat, unitree}, the recording platform (see Additional platforms). The remaining fields:

bag_id, session, start_time, split, is_daytime, degraded, has_seg, duration_s, n_lidar_frames, n_rgb_frames, n_imu_samples, n_events_left, n_events_right, n_gps_fixes, n_gps_valid, gps_quality, gps_lat_min/max, gps_lon_min/max, mean_speed_mph, idle_fraction, distance_m, rgb_cal_id, imu_cal_id, lidar_cal_id, sensor_dropout

• —gps_quality ∈ {RTK_fixed_cm, float_dm, single_m, no_fix, absent}, 67 / 62 / 239 / 2 / 1 (RTK-fixed ≈1–3 cm, float ≈0.1–0.5 m, single ≈0.5–3 m).
• —sensor_dropout: null, or sensor:seconds[;sensor:seconds]

Download

The full dataset is 8+ TB. The `download_octosense.py` helper (in the GitHub repo) selects by platform, sequence, and modality, and can skip the raw event streams (events.h5, ~78% of each sequence's bytes):

pip install huggingface_hub

# all car sequences (everything)
python download_octosense.py --platform car --out ./octosense

# the same, but skip the raw event streams (events.h5, ~78% of the bytes)
python download_octosense.py --platform car --no-events

# one sequence, only the core LiDAR/IMU/GPS h5 + depth + segmentation
python download_octosense.py --sequence rosbag2_2026_01_04-13_51_24 --modalities data,depth,seg

--modalities picks from data, events, captions, depth, seg, rgb, ir.

Or pull files directly with huggingface_hub:

from huggingface_hub import snapshot_download
snapshot_download("anthonytec2/OctoSense", repo_type="dataset",
                  allow_patterns="car/sess7/**")          # one session

Loading

import h5py, hdf5plugin, numpy as np, pyarrow.parquet as pq
meta = pq.read_table("metadata.parquet").to_pydict()       # per-sequence table
with h5py.File("<platform>/<session>/<bag_id>/data.h5") as f:
    lidar = f["ouster/range_pcl"][:]
    odom = f["ouster/odom/map_T_lidart"][:]

# RGB / IR frames, native-res H.265 mp4 decoded with torchcodec.
from torchcodec.decoders import VideoDecoder
dec = VideoDecoder("<platform>/<session>/<bag_id>/img_left.mp4")      # 1920x1456; len(dec) == n_rgb_frames
rgb_k = dec[k]                                             # (3, H, W) uint8 tensor at frame k
# RGB runs ≈100 Hz vs LiDAR ≈10 Hz, align by TIMESTAMP, not shared index:
with h5py.File("<platform>/<session>/<bag_id>/data.h5") as f:
    img_t = f["img/left/t"][:]                            # seconds, same main clock
    lid_t = f["ouster/t"][:]
k = int(np.argmin(np.abs(img_t - lid_t[j])))              # RGB frame nearest LiDAR frame j
# img_right.mp4 / img_infrared.mp4 decode the same way (IR is grayscale-as-video, ≈50 Hz).

# Depth + semantic GT, one frame per LiDAR scan, in the rectified left-RGB image:
with h5py.File("<platform>/<session>/<bag_id>/rgb_left_rect_depth.h5") as f:
    depth_m = f["depth_cm"][j] / 100.0                     # (1456,1920); 0 = invalid (j = LiDAR scan)
with h5py.File("<platform>/<session>/<bag_id>/rgb_left_rect_semantic.h5") as f:   # day sequences only
    seg     = f["semantic"][j]                             # (1456,1920) uint8, class 0..18
    rgb_idx = f["left_img_indices"][j]                     # matching img_left.mp4 frame for scan j

# flow is derived, not stored — for LiDAR scan j, pass odom as poses (map_T_lidart).
# Flow is from scan j -> j+gap; gap is read from the depth h5 attr `flow_gap` (default 2).
from derive_flow import derive_flow_from_h5
flow_i16 = derive_flow_from_h5("<platform>/<session>/<bag_id>/rgb_left_rect_depth.h5", j, poses=odom)
flow_uv = flow_i16.astype(np.float32) / 8.0               # (2,H,W) px (invalid where flow_i16 == -32768)

Statistics

382 sequences across 3 platforms:

• —car: 371 sequences · 59 hrs · 2,474 km · 8.43 TB · 303 day / 68 night · 4 degraded · segmentation on 303 · GPS: 67 RTK / 62 float / 239 single / 2 no-fix / 1 absent.
• —boat: 9 sequences · ≈1.2 hrs · GPS: 8 RTK / 1 absent
• —unitree: 2 sequences · ≈15 min · ≈0.8 km traversed · quadruped joint state

Notes/Known limitations

• —Optical flow is derived solely from ego-motion: rigid camera-motion reprojection of LiDAR depth; it does not capture independent motion of dynamic objects.
• —Seg is day-only (303 sequences); nighttime sequences have no segmentation labels.
• —No color correction was performed on the RGB frames, so colors can look raw or cast. You can apply a gray-world white-balance correction if you need more natural-looking colors.
• —IMU compensated vs raw. VectorNav factory-calibrates each unit (bias/scale/axis-misalignment + temperature applied). In this dataset accel ≡ accel_raw; ang_vel additionally has the on-board EKF's real-time gyro-bias estimate removed.
• —5 sequences have a >10 s sensor dropout (see sensor_dropout): 3 with the event camera off early (20–43 s), 1 GPS+CAN, 1 CAN. 3 sequences have no usable GPS (no_fix/absent), and 5 sequences have no `fused_traj` (GPS too sparse / low-quality for the pose-graph fusion).
• —IR & GPS are PPS-synced indirectly, through the IMU PPS-system clock calibration.
• —IR calibration is best-effort. The infrared intrinsics/extrinsics were calibrated from existing data without a heated target board. They are estimated from the markerboard's 4 corners alone, then manually refined. Treat the IR calibration as approximate.
• —LiDAR clock is PPS-stepped, not Kalman-aligned. Unlike the other streams, the Ouster clock is not actively Kalman-smoothed onto the main clock; its internal clock only advances its whole-second counter when the PPS edge arrives.
• —IMU is noisy from vehicle vibration. Both IMUs pick up road and engine vibration; the VectorNav is soft-mounted to dampen it, but residual vibration noise remains in the accelerometer / gyro signals.
• —IR frame rate occasionally dips below 50 Hz. The infrared camera is not run as a composable node, so under load it sometimes falls below its nominal 50 Hz capture rate (gaps visible in infrared/t).
• —RKO-LIO cold-start transient. The estimator's initial phase can occasionally be jerky, at the start of a sequence the LIO briefly reports ≈zero motion, then "catches up" with a jump once it converges → jerky roll/pitch (occasionally z) over the first few meters of motion (upstream RKO-LIO issue #139).
• —LiDAR deskew is a constant-motion approximation. Each ~100 ms sweep is motion-compensated by RKO-LIO using the average body acceleration a and average angular velocity ω over the sweep, a point at offset dt from the scan reference time is warped by exp([ v·dt + ½·a·dt², ω·dt ]) (constant-acceleration translation + constant-angular-velocity rotation). Because a/ω are held constant across the sweep, rapid intra-sweep motion (high jerk, sharp turns, potholes) leaves some residual skew; a is Kalman-filtered + jerk-bounded to limit this but cannot fully remove it.
• —LiDAR noise can leak into the depth GT. The depth ground truth is projected directly from the raw Ouster returns, so sensor noise (stray or spurious returns from rain, snow, fog, dust, retroreflectors (see LiDAR ghosts & blooming), or specular/multi-path reflections) can survive into our depth ground truth as a small number of erroneous points.
• —Platform Bounce. On a few sequences the SeaSuckers loosened, resulting in vertical motion (bounce) of the platform.
• —Raw ROS 2 bags are currently unreleased. If you require the original ROS 2 bag recordings, feel free to contact pratikac@upenn.edu.

We have taken great care to perform data quality checks on this data. That said, some issues at this scale may slip through, so should you find any examples of gross desynchronization, please report them and we can take a look.

Citation

@misc{bisulco2026octosense,
  title        = {{OctoSense}: Self-Supervised Learning for Multimodal Robot Perception},
  author       = {Bisulco, Anthony and Wang, Jeremy and Daniilidis, Kostas and Balestriero, Randall and Chaudhari, Pratik},
  year         = {2026},
  howpublished = {Preprint},
}

License

Released under the MIT License: free to use, modify, and redistribute with attribution; provided "as is" without warranty. If you use OctoSense, please cite the paper (see Citation above).