ys-qu/found-rl_dataset

Found-RL Dataset: Demonstration Data for VLM Fine-tuning

🚗 Dataset Overview

This dataset contains large-scale demonstration data collected from the CARLA simulator, designed to fine-tune Vision-Language Models (VLMs) for autonomous driving tasks. It serves as the data foundation for the paper "Found-RL: Foundation Model-Enhanced Reinforcement Learning for Autonomous Driving".

The dataset comprises approximately 1.374 million state-action transitions collected across three diverse benchmarks using expert policies.

• —📄 Paper: Found-RL: foundation model-enhanced reinforcement learning for autonomous driving
• —💻 Code: https://github.com/ys-qu/found-rl
• —📦 Format: Compressed .tar.gz archive

📊 Dataset Statistics & Composition

We collected demonstration data on three primary benchmarks to ensure diversity in driving scenarios. The total dataset consists of ~1.37M transitions.

🛠 Data Collection Methodology

1. Expert Policies

• —Leaderboard Benchmark: Data was collected using the Roach PPO expert policy (Zhang et al., 2021).
• —NoCrash & Challenge Benchmarks: Data was collected using the Autopilot roaming expert policy.

2. Constraints & Filtering

To ensure high-quality training data for VLM fine-tuning, the following constraints were applied during collection:

• —Maximum Duration: The maximum duration for each episode was set to 300 seconds.
• —Collision Filtering: A terminal step filtering rule was applied. A short segment of steps immediately preceding a collision event was discarded, ensuring the dataset contains only the valid, safe portion of each episode.

3. Usage

This data is intended to be used with open-source frameworks (e.g., open_clip, LLaVA codebases) to fine-tune VLMs, providing them with expert-level driving understanding.

If you use this dataset in your research, please cite our paper:

@misc{qu2026foundrl,
      title={Found-RL: foundation model-enhanced reinforcement learning for autonomous driving}, 
      author={Yansong Qu and Zihao Sheng and Zilin Huang and Jiancong Chen and Yuhao Luo and Tianyi Wang and Yiheng Feng and Samuel Labi and Sikai Chen},
      year={2026},
      eprint={2602.10458},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2602.10458}, 
}