AutowareFoundation/lidar_apollo_instance_segmentation

Apollo LiDAR Instance Segmentation for Autoware (lidar_apollo_instance_segmentation)

CNN-based LiDAR instance segmentation models, used by the `autoware_lidar_apollo_instance_segmentation` node in Autoware.

The models originate from the Baidu Apollo 3D obstacle perception stack (CNN segmentation). The original Apollo caffemodels were converted to ONNX for use in Autoware, and the node builds a TensorRT engine from the ONNX file on first launch. The node rasterizes the input point cloud into a bird's-eye-view feature map, runs the network, and clusters the per-cell predictions into labeled obstacle instances.

Model overview

Variants in this repository

One ONNX model per supported LiDAR sensor, selected via the model launch argument:

The per-variant parameter files (vlp-16.param.yaml, hdl-64.param.yaml, vls-128.param.yaml) live in the consuming package under config/, not in this repository. The values above come from those files. Supported LiDARs are the Velodyne 16, 64, and 128 beam sensors, but the package README notes that other LiDARs such as the Velodyne 32 can also be used with good accuracy.

Files

TensorRT engines are not distributed here. TensorRT engines are specific to the GPU architecture and TensorRT version they are built on and are not portable, so Autoware builds them locally from the ONNX files on first launch (or via build_only:=true).

Inputs and outputs (as used by the node)

Input: input/pointcloud (sensor_msgs/msg/PointCloud2), point cloud data from LiDAR sensors (default remap: /sensing/lidar/pointcloud).

Outputs:

• —output/labeled_clusters (tier4_perception_msgs/msg/DetectedObjectsWithFeature): detected objects with labeled point cloud clusters.
• —debug/instance_pointcloud (sensor_msgs/msg/PointCloud2): segmented point cloud for visualization.

Pre-processing (bird's-eye-view feature map generation) and post-processing (2D clustering of the network output into obstacle instances, score thresholding via score_threshold) run in the node, not in the ONNX graphs.

Usage in Autoware

The node expects these artifacts under ~/autoware_data/ml_models/lidar_apollo_instance_segmentation/ and launches with, e.g.:

ros2 launch autoware_lidar_apollo_instance_segmentation lidar_apollo_instance_segmentation.launch.xml \
  model:=model_128 \
  data_path:=$HOME/autoware_data/ml_models

model:=model_16, model_64, or model_128 selects vlp-16.onnx, hdl-64.onnx, or vls-128.onnx respectively, together with the matching parameter file from the package's config/ directory. Add build_only:=true to build the TensorRT engine from the ONNX as a one-off pre-task. See the package README for the full parameter reference.

Training

There is no training code for these models. The weights were trained by the Baidu Apollo project and released as caffemodels, which were later converted to ONNX for Autoware. Training datasets, schedules, and metrics are not publicly documented.

Origin and related projects:

• —Apollo autonomous driving platform (original caffemodels and CNN segmentation design): <https://github.com/ApolloAuto/apollo>
• —Initial Autoware integration of the Apollo CNN segmentation detector: <https://github.com/k0suke-murakami/autoware_perception>
• —TensorRT wrapper library historically used by the node (current code uses autoware_tensorrt_common): <https://github.com/lewes6369/tensorRTWrapper>

The original caffemodel sources (pinned Apollo commits) are listed in the package README:

• —VLP-16: modules/perception/production/data/perception/lidar/models/cnnseg/velodyne16/deploy.caffemodel (Apollo commit 88bfa5a)
• —HDL-64: modules/perception/production/data/perception/lidar/models/cnnseg/velodyne64/deploy.caffemodel (Apollo commit 88bfa5a)
• —VLS-128: modules/perception/production/data/perception/lidar/models/cnnseg/velodyne128/deploy.caffemodel (Apollo commit 91844c8)

Provenance

The v1.0 tag corresponds to the exact file set previously served from the unversioned awf.ml.dev.web.auto path.

Limitations

• —Each ONNX file is tuned for a specific Velodyne sensor (16, 64, or 128 beams); other LiDARs may work with good accuracy but are not the intended configuration.
• —Only the classes listed above are produced; other road users are reported as UNKNOWN.
• —No training code or training data documentation exists, so the models cannot be retrained or fine-tuned from public sources.

References

• —Apollo 3D Obstacle Perception description: <https://github.com/ApolloAuto/apollo/blob/r7.0.0/docs/specs/3dobstacleperception.md>
• —Consuming Autoware package: <https://github.com/autowarefoundation/autowareuniverse/tree/main/perception/autowarelidarapolloinstance_segmentation>

Legal notice

The model weights are derived from the Baidu Apollo project, which is licensed under the Apache License 2.0. The consuming Autoware package additionally incorporates code from the Apollo project (Apache-2.0), the tensorRTWrapper library (MIT), and the autoware_perception integration (Apache-2.0); see the package README for the full license texts.