AutowareFoundation/tensorrt_vad

VAD for Autoware (tensorrt_vad)

End-to-end autonomous driving model (variant carla_tiny) used by the `autoware_tensorrt_vad` node in Autoware.

The model follows the VAD (Vectorized Autonomous Driving) [1] architecture and was optimized for TensorRT deployment with NVIDIA's DL4AGX workflow. It replaces the traditional localization, perception, and planning modules with a single neural network: 6 surround-view camera images go in, an ego trajectory comes out, with predicted objects and map elements as auxiliary outputs.

Simulation only. This model is trained exclusively on CARLA simulation data from the Bench2Drive benchmark [2]. It is intended for use with the CARLA simulator and is not suitable for real-vehicle deployment.

Model overview

The network is split into three ONNX sub-models, matching how the node consumes them:

1. 1.Backbone (vad-carla-tiny_backbone.onnx): image feature extraction from the 6 camera images, run in FP16 by default.
2. 2.Head (vad-carla-tiny_head.onnx): temporal planning head that consumes the backbone features (mlvl_feats.0) together with previous-frame history, run in FP32 by default.
3. 3.Head, no history (vad-carla-tiny_head_no_prev.onnx): planning head used for the first frame, when no temporal history exists yet, run in FP32 by default.

Key architecture parameters (from vad-carla-tiny.param.json): 640x384 model input per camera with ImageNet RGB normalization, 106x200 (height x width) BEV grid with 256 feature channels, 3 transformer decoder layers, 300 object queries (9 classes, 6 predicted trajectory modes over 6 timesteps), 100 map queries (6 classes, 20 points per polyline), and 6 ego planning commands over 6 timesteps.

Files

TensorRT engines are not distributed here. TensorRT engines are specific to the GPU architecture and TensorRT version they are built on and are not portable, so Autoware builds them locally from the ONNX files on first launch and caches them next to the ONNX files.

Inputs and outputs (as used by the node)

Inputs

Outputs

The camera order is fixed by the training data and must not be changed. The node remaps the CARLA object classes to Autoware classes via object_class_remapper_carla_tiny.param.yaml (car/van to CAR, truck to TRUCK, bicycle to BICYCLE, pedestrian to PEDESTRIAN, static classes and others to UNKNOWN).

Usage in Autoware

Autoware downloads these artifacts to ~/autoware_data/ml_models/vad/ (note: the local directory is named vad, not tensorrt_vad), and the node loads them from $(var model_path)/v0.1/ as configured in config/vad_carla_tiny.param.yaml.

After setting up CARLA following the autoware_carla_interface instructions, launch the end-to-end VAD system with:

ros2 launch autoware_launch e2e_simulator.launch.xml \
  map_path:=$HOME/autoware_data/maps/Town01 \
  vehicle_model:=sample_vehicle \
  sensor_model:=carla_sensor_kit \
  simulator_type:=carla \
  use_e2e_planning:=true

The node itself is started via vad_carla_tiny.launch.xml, which defaults model_path to $HOME/autoware_data/ml_models/vad. On first run it builds the TensorRT engines from the ONNX files, optimized for the local GPU, and caches them for subsequent runs. See the package README for the full parameter reference.

Training

The model was trained on CARLA simulation data from the Bench2Drive benchmark and deployed with NVIDIA's DL4AGX TensorRT workflow:

• —Architecture: VAD (Jiang et al., arXiv:2303.12077)
• —Training data / benchmark: Bench2Drive (Jia et al., arXiv:2406.03877)
• —Deployment / TensorRT optimization: DL4AGX

Initial release (v0.1) date: 2025-11-04. Further training details (epochs, exact frame counts) are not publicly documented.

Limitations

• —Simulation-only training: the model is trained exclusively on CARLA simulator data, which does not capture the full complexity and variability of real-world driving. Do not deploy it on a real vehicle.
• —No dynamic mission control: the current implementation lacks a high-level command interface, so the model cannot switch driving behaviors (for example from "follow lane" to "turn right at the next intersection") at runtime. The default command is LANE_FOLLOW.

Provenance

Citation

@inproceedings{jiang2023vad,
  title     = {VAD: Vectorized Scene Representation for Efficient Autonomous Driving},
  author    = {Jiang, Bo and Chen, Shaoyu and Xu, Qing and Liao, Bencheng and Chen, Jiajie and Zhou, Helong and Zhang, Qian and Liu, Wenyu and Huang, Chang and Wang, Xinggang},
  booktitle = {ICCV},
  year      = {2023}
}
@article{jia2024bench2drive,
  title   = {Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving},
  author  = {Jia, Xiaosong and Yang, Zhenjie and Li, Qifeng and Zhang, Zhiyuan and Yan, Junchi},
  journal = {arXiv preprint arXiv:2406.03877},
  year    = {2024}
}

References

• —[1] Jiang et al., "VAD: Vectorized Scene Representation for Efficient Autonomous Driving", arXiv:2303.12077, 2023. Code: <https://github.com/hustvl/VAD>
• —[2] Jia et al., "Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving", arXiv:2406.03877, 2024. Code: <https://github.com/Thinklab-SJTU/Bench2Drive>
• —[3] NVIDIA DL4AGX, TensorRT optimization for autonomous driving workloads: <https://github.com/NVIDIA/DL4AGX>