VLyb/VLABench

VLABench VLM Evaluation Dataset

This dataset is the VLM evaluation split of VLABench, prepared for reproducible VLABench evaluation with PhysBrainEvalKit.

Source

• —Project page: <https://vlabench.github.io/>
• —Paper: <https://arxiv.org/abs/2412.18194>
• —Official code: <https://github.com/OpenMOSS/VLABench>

Directory layout

The dataset is organized by evaluation dimension and subtask:

vlm_evaluation_v1.0/
├── CommenSence/
├── Complex/
├── M&T/
├── PhysicsLaw/
├── Semantic/
└── Spatial/

Each subtask contains evaluation episodes, typically example0 through example99:

<dimension>/<subtask>/example0/
├── input/
│   ├── input.png
│   └── input_mask.png
├── instruction/
│   └── input_instruction.txt
├── output/
│   └── operation_sequence.json
└── env_config/

The PhysBrainEvalKit evaluator requires input/input.png, input/input_mask.png, instruction/input_instruction.txt, and output/operation_sequence.json. The optional env_config/ directory contains episode configuration files for users who want to reproduce the underlying simulation environment; it is not required for VLM scoring.

Use with PhysBrainEvalKit

Download or clone this dataset directory, then set its local path:

export VLABENCH_DATASET_PATH=/path/to/vlm_evaluation_v1.0

Run the VLABench benchmark from the PhysBrainEvalKit repository:

python eval_vlabench.py \
  --model_path /path/to/qwen3_vl_model \
  --model_name qwen3-vl \
  --backbone qwen3 \
  --backend hf \
  --dataset_path "$VLABENCH_DATASET_PATH"

The standard launcher uses the same path automatically when VLABENCH_DATASET_PATH is set:

bash scripts/eval_qwen3vl.sh \
  --model-path /path/to/qwen3_vl_model \
  --model-name qwen3-vl \
  --output-base /path/to/results \
  --gpus 0,1 \
  --only VLABench

Citation

If you use VLABench, please cite the original paper:

@misc{zhang2024vlabench,
  title={VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks},
  author={Shiduo Zhang and Zhe Xu and Peiju Liu and Xiaopeng Yu and Yuan Li and Qinghui Gao and Zhaoye Fei and Zhangyue Yin and Zuxuan Wu and Yu-Gang Jiang and Xipeng Qiu},
  year={2024},
  eprint={2412.18194},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2412.18194}
}

License and redistribution

VLABench data and media remain subject to the license and redistribution terms of the original project. Review the upstream repository before publishing a mirror or redistributing the files.