xlzhou126/SpaceDG-Bench

SpaceDG-Bench

**🌐 Homepage** | **📖 arXiv** | **💻 GitHub**

SpaceDG-Bench is a human-verified benchmark designed to evaluate the spatial intelligence of Multimodal Large Language Models (MLLMs) under visual degradation. It contains 1,102 questions spanning 11 reasoning categories and 9 visual degradation types (such as motion blur, low light, adverse weather, lens distortion, and compression artifacts), yielding over 10K VQA instances.

The benchmark is part of the SpaceDG project, which introduces the first large-scale dataset for degradation-aware spatial understanding, utilizing a physically grounded synthesis engine embedded into 3D Gaussian Splatting (3DGS) rendering.

Data files

• —data/spacedg_bench-*-of-*.parquet: the dataset shards (6-way split, size-balanced). They contain images (multi-image, embedded bytes) and basic metadata columns.
• —spacedg_bench.tsv: question/answer/metadata table. The image_path field stores a Python-style list of relative image paths (e.g., defocus/.../*.jpg), typically relative to LMUData/images/spacedg_bench/.
• —dataset_infos.json: Hugging Face metadata that marks images as Sequence(Image) so the Dataset Viewer can render thumbnails instead of showing truncated bytes.
• —prepare_data.py: Decode the parquet shards (or fallback to spacedg_bench.parquet) and convert it to the format powered by EASI for evaluation.
• —repack_parquet_shards.py: Repack TSV + local images into 6 parquet shards. Use --balance bytes to keep shard sizes close.

Parquet schema (high level)

• —id (int64): sample id
• —images (sequence[image]): a list of images for the sample
• —question (string)
• —answer (string)
• —question_type (string)
• —task_group (string)
• —degradation_type (string): inferred degradation category (e.g., defocus)

Sample Usage (Evaluation)

To evaluate a model using the provided evaluation framework (VLMEvalKit), you can use the following command:

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 torchrun run.py \
  --model InternVL3_5-8B \
  --data spacedg_bench \
  --mode all \
  --work-dir ../outputs_spacedg \
  --reuse

Citation

@article{zhou2026spacedg,
  title={SpaceDG: Benchmarking Spatial Intelligence under Visual Degradation},
  author={Xiaolong Zhou and Yifei Liu and Ziyang Gong and Jiarui Li and Qiyue Zhao and Muyao Niu and Yuanyuan Gao and Le Ma and Xue Yang and Hongjie Zhang and Zhihang Zhong},
  journal={arXiv preprint arXiv:2605.22536},
  year={2026}
}