AutowareFoundation/tensorrt_yolox

1---2license: apache-2.03pipeline_tag: object-detection4tags:5  - autoware6  - ros27  - autonomous-driving8  - camera9  - 2d-object-detection10  - yolox11  - semantic-segmentation12  - traffic-light13  - tensorrt14  - onnx15  - int816---17 18# YOLOX for Autoware (`tensorrt_yolox`)19 202D object detection (and optional semantic segmentation) models for camera images, used by the21[`autoware_tensorrt_yolox`](https://github.com/autowarefoundation/autoware_universe/tree/main/perception/autoware_tensorrt_yolox)22node in [Autoware](https://github.com/autowarefoundation/autoware).23 24The models follow the **YOLOX** [1] architecture. They are exported as ONNX so they can be deployed across25hardware; Autoware builds the TensorRT engine from the ONNX file on first launch. The same node binary consumes26every model in this repository: the camera object detection pipeline uses the detection and detection+segmentation27models, and a separate node instance runs the whole-image traffic light detector that feeds the traffic light28recognition pipeline.29 30## Model overview31 32| | |33| --- | --- |34| Task | 2D object detection from a camera image; one variant additionally outputs a semantic segmentation mask, one variant detects traffic lights on the whole image |35| Architecture | YOLOX (tiny and s-based variants); the semantic segmentation variant uses a multi-header structure on a YOLOX-s base |36| Detected classes | `CAR`, `PEDESTRIAN`, `BUS`, `TRUCK`, `BICYCLE`, `MOTORCYCLE` (detection models) or `UNKNOWN`, `CAR_TRAFFIC_LIGHT`, `PEDESTRIAN_TRAFFIC_LIGHT` (traffic light detector) |37| Runtime | TensorRT (FP32 / FP16 / INT8) via the `autoware_tensorrt_yolox` ROS 2 node |38| Format | ONNX plus INT8 calibration tables (Autoware builds the TensorRT engine locally on first launch) |39| License | Apache-2.0 |40 41Labels listed in the model's label file that are not in the node's known set are reported as `UNKNOWN`.42`yolox-tiny.onnx` has an `EfficientNMS_TRT` module attached after the network to accelerate non-maximum43suppression; the module contains fixed `score_threshold` and `nms_threshold` values, so those node parameters44are ignored for models that include it.45 46## Model families in this repository47 48| Family | Files | Original source path | Consuming launch file |49| --- | --- | --- | --- |50| Generic detection | `yolox-tiny.onnx`, `yolox-sPlus-opt.onnx`, `yolox-sPlus-opt.EntropyV2-calibration.table`, `label.txt` | unversioned root of the `awf.ml.dev.web.auto` model store (not browsable as a directory; direct per-file URLs in Provenance) | `yolox_tiny.launch.xml` (tiny); s-Plus-opt selectable via `model_path` |51| Pseudo-finetuned detection | `yolox-sPlus-T4-960x960-pseudo-finetune.onnx`, `yolox-sPlus-T4-960x960-pseudo-finetune.EntropyV2-calibration.table` | `https://awf.ml.dev.web.auto/perception/models/object_detection_yolox_s/v1/` | `yolox_s_plus_opt.launch.xml` (detection-only option for `model_path`) |52| Detection + semantic segmentation (16 classes) | `yolox-sPlus-opt-pseudoV2-T4-960x960-T4-seg16cls.onnx`, `yolox-sPlus-opt-pseudoV2-T4-960x960-T4-seg16cls.EntropyV2-calibration.table`, `semseg_color_map.csv` | `https://awf.ml.dev.web.auto/perception/models/object_detection_semseg_yolox_s/v1/` | `yolox_s_plus_opt.launch.xml` (default `model_path`) |53| Whole-image traffic light detector | `yolox_s_car_ped_tl_detector_960_960_batch_1.onnx`, `yolox_s_car_ped_tl_detector_960_960_batch_1.EntropyV2-calibration.table`, `car_ped_tl_detector_labels.txt` | `https://awf.ml.dev.web.auto/perception/models/tl_detector_yolox_s/v1/` | `yolox_traffic_light_detector.launch.xml` (separate node instance feeding the traffic light recognition pipeline) |54 55Notes on the families:56 57- `yolox-sPlus-opt-pseudoV2-T4-960x960-T4-seg16cls` is a multi-header model based on YOLOX-s, tuned to detect58  more accurately than `yolox-tiny` at almost comparable execution speed. Besides detection it outputs a59  semantic segmentation mask used for point cloud filtering. Recommended settings: `precision:=int8`,60  `calibration_algorithm:=Entropy`, `clip_value:=6.0` (these are the defaults in `yolox_s_plus_opt.param.yaml`).61- `yolox-sPlus-T4-960x960-pseudo-finetune` is the detection-only alternative offered by the same launch file.62- The traffic light detector localizes car and pedestrian traffic lights on the full camera image; its labels63  come from `car_ped_tl_detector_labels.txt` (`BACKGROUND`, `traffic_light`, `pedestrian_traffic_light`).64- `label.txt` for the detection models contains `UNKNOWN`, `CAR`, `TRUCK`, `BUS`, `BICYCLE`, `MOTORBIKE`,65  `PEDESTRIAN`, `ANIMAL`.66 67The semantic segmentation mask is a gray image where each pixel holds a class index; `semseg_color_map.csv`68maps the 16 indices to names and RGB colors for visualization (others, building (spelled `buildling` in the69shipped CSV), wall, obstacle, traffic_light,70traffic_sign, person, vehicle, bike, road, sidewalk, roadPaint, curbstone, crosswalk_others, vegetation, sky).71 72## Files73 74| File | Description |75| --- | --- |76| `yolox-tiny.onnx` | YOLOX-tiny detection model with `EfficientNMS_TRT` attached |77| `yolox-sPlus-opt.onnx` | YOLOX-s based optimized detection model |78| `yolox-sPlus-opt.EntropyV2-calibration.table` | INT8 calibration table for `yolox-sPlus-opt` |79| `yolox-sPlus-T4-960x960-pseudo-finetune.onnx` | YOLOX-s based detection model, pseudo-label finetuned, 960x960 input |80| `yolox-sPlus-T4-960x960-pseudo-finetune.EntropyV2-calibration.table` | INT8 calibration table for the pseudo-finetuned model |81| `yolox-sPlus-opt-pseudoV2-T4-960x960-T4-seg16cls.onnx` | Multi-header detection + 16-class semantic segmentation model, 960x960 input |82| `yolox-sPlus-opt-pseudoV2-T4-960x960-T4-seg16cls.EntropyV2-calibration.table` | INT8 calibration table for the detection + segmentation model |83| `yolox_s_car_ped_tl_detector_960_960_batch_1.onnx` | Whole-image traffic light detector (YOLOX-s, 960x960, batch 1) |84| `yolox_s_car_ped_tl_detector_960_960_batch_1.EntropyV2-calibration.table` | INT8 calibration table for the traffic light detector |85| `label.txt` | Class labels for the detection models |86| `car_ped_tl_detector_labels.txt` | Class labels for the traffic light detector |87| `semseg_color_map.csv` | Semantic segmentation class index to name and RGB color map |88| `deploy_metadata.yaml` | Deployment metadata recording the artifact version of this repository |89 90> **TensorRT engines are not distributed here.** TensorRT engines are specific to the GPU architecture and91> TensorRT version they are built on and are not portable, so Autoware builds them locally from the ONNX files92> on first launch (or via `build_only:=true`). Engines are saved next to the ONNX files with an `.engine`93> extension and reused on subsequent runs; the first build typically takes 10 to 20 minutes. The94> `EntropyV2-calibration.table` files are used by the node when running with `precision:=int8`.95 96## Inputs and outputs (as used by the node)97 98**Input**: `~/in/image` (`sensor_msgs/msg/Image`), the camera image.99 100**Outputs**:101 102- `~/out/objects` (`tier4_perception_msgs/msg/DetectedObjectsWithFeature`): detected objects or traffic lights103  with 2D bounding boxes.104- `~/out/image` (`sensor_msgs/msg/Image`): input image with 2D bounding boxes drawn, for visualization.105- `~/out/mask` (`sensor_msgs/msg/Image`): semantic segmentation mask (semantic segmentation model only).106- `~/out/color_mask` (`sensor_msgs/msg/Image`): colorized segmentation mask for visualization (semantic107  segmentation model only).108 109## Usage in Autoware110 111The node reads these artifacts from `$HOME/autoware_data/ml_models/tensorrt_yolox/` by default and launches112with, e.g.:113 114```bash115# Detection + semantic segmentation (default), or detection-only via model_path116ros2 launch autoware_tensorrt_yolox yolox_s_plus_opt.launch.xml117 118# Lightweight detection119ros2 launch autoware_tensorrt_yolox yolox_tiny.launch.xml120 121# Whole-image traffic light detection122ros2 launch autoware_tensorrt_yolox yolox_traffic_light_detector.launch.xml123```124 125Add `build_only:=true` to build the TensorRT engine from the ONNX as a one-off pre-task. See the126[package README](https://github.com/autowarefoundation/autoware_universe/tree/main/perception/autoware_tensorrt_yolox)127for the full parameter reference, including precision selection (`fp32`, `fp16`, `int8`) and INT8 calibration128options.129 130## Training131 132The models are based on the official YOLOX implementation. The training datasets, schedules, and evaluation133metrics of the T4 finetuned variants are not publicly documented.134 135- YOLOX (architecture and training framework): <https://github.com/Megvii-BaseDetection/YOLOX>136- trt-yoloXP (TIER IV YOLOX extensions): <https://github.com/tier4/trt-yoloXP>137- yolox_onnx_modifier (embeds `EfficientNMS_TRT` into exported ONNX): <https://github.com/wep21/yolox_onnx_modifier>138 139The consuming package documents how to export custom YOLOX models to ONNX (plain or with `EfficientNMS_TRT`)140in its README.141 142## Provenance143 144Original hosting before migration to Hugging Face (tag `v1.0` bundles all four families):145 146| Family | Original source |147| --- | --- |148| Generic detection | unversioned root of the model store; the root itself is not browsable (directory listing is disabled), so the direct per-file URLs are: <https://awf.ml.dev.web.auto/perception/models/yolox-tiny.onnx>, <https://awf.ml.dev.web.auto/perception/models/yolox-sPlus-opt.onnx>, <https://awf.ml.dev.web.auto/perception/models/yolox-sPlus-opt.EntropyV2-calibration.table>, <https://awf.ml.dev.web.auto/perception/models/label.txt> |149| Pseudo-finetuned detection | `https://awf.ml.dev.web.auto/perception/models/object_detection_yolox_s/v1/` |150| Detection + semantic segmentation | `https://awf.ml.dev.web.auto/perception/models/object_detection_semseg_yolox_s/v1/` |151| Traffic light detector | `https://awf.ml.dev.web.auto/perception/models/tl_detector_yolox_s/v1/` |152 153## Limitations154 155- Detection models output only the classes listed above; other road users are reported as `UNKNOWN`.156- `label.txt` is incompatible with models that output COCO labels (e.g. models from the official YOLOX157  repository); those need their own label file.158- Fixed `score_threshold` and `nms_threshold` are baked into models with `EfficientNMS_TRT`159  (e.g. `yolox-tiny.onnx`); the node parameters of the same names have no effect for them.160- Training data details of the T4 finetuned variants are not publicly documented, so accuracy on sensor setups161  and environments different from those used for training is not characterized here.162 163## Citation164 165```bibtex166@article{yolox2021,167  title   = {YOLOX: Exceeding YOLO Series in 2021},168  author  = {Ge, Zheng and Liu, Songtao and Wang, Feng and Li, Zeming and Sun, Jian},169  journal = {arXiv preprint arXiv:2107.08430},170  year    = {2021}171}172```173 174## References175 176- [1] Ge et al., "YOLOX: Exceeding YOLO Series in 2021", arXiv:2107.08430, 2021.177- Megvii-BaseDetection/YOLOX: <https://github.com/Megvii-BaseDetection/YOLOX>178- tier4/trt-yoloXP: <https://github.com/tier4/trt-yoloXP>179- wep21/yolox_onnx_modifier: <https://github.com/wep21/yolox_onnx_modifier>180