Wenliang04/HRScene

<div align="center"><h5>HRScene - High Resolution Image Understanding</h5></div>

<a href="https://yszh8.github.io/hrscene">🌐 Homepage</a> | <a href="https://huggingface.co/datasets/Wenliang04/HRScene">🤗 Dataset</a> | <a href="https://arxiv.org/pdf/2504.18406">📖 arXiv</a> | <a href="https://github.com/psunlpgroup/HRScene">GitHub</a>

⭐ About HRScene

<p>We introduce <strong>HRScene</strong>, a novel unified benchmark for HRI understanding with rich scenes. HRScene incorporates 25 real-world datasets and 2 synthetic diagnostic datasets with resolutions ranging from <strong>1,024 × 1,024 to 35,503 × 26,627</strong>. HRScene is collected and re-annotated by 10 graduate-level annotators, covering <strong>25 scenarios</strong>, ranging from microscopic and radiology images to street views, long-range pictures, and telescope images. It includes high-resolution images of real-world objects, scanned documents, and composite multi-image.</p> <img src="example.png" alt="Dataset Sample Image" class="resized-image" style="max-width:100%; height:auto; display:block;"> <p> Some examples of HRScene. Blue ones are diagnostic datasets, and purple ones are real-world datasets.</p>

<p>HRScene consists of 7,073 samples, divided into three splits:</p>

Val contains 750 samples. These samples are identical to human-annotated ones, designed for fine-grained validation of the users' VLM settings.

Testmini comprises 1,000 samples, picked from each HRScene real-world dataset, intended for rapid model development evaluation or for those with limited computing resources.

Test features the remaining 5,323 samples for standard evaluation. Notably, the answer labels for test will not be publicly released to facilitate fair evaluation. Instead, we maintain an online evaluation platform for user submissions.

<p align="center"> <img src="piechart.png" width="60%"> <br> </p>

📖 Dataset Usage

Data Downloading

By using our pipeline, you dont need to download the dataset manually.

For whitebackground and complexgrid, you only need to set the dataset_name for the tester like we did in the Diagnosis/example.py.

tester = DiagnosisTester(model=model, dataset_name="complexgrid_3x3", num_samples=150)

For realworld, you need to set the dataset_name and split for the tester like we did in the RealWorld/example.py.

tester = RealWorldTester(model=model, dataset_name="realworld_combined", split="test")

Or you wanna download the dataset manually, you can use the following code:

from datasets import load_dataset


# for whitebackground and complexgrid, we only have 'test' split
dataset = load_dataset("Wenliang04/HRScene", "whitebackground_1x1")
for sample in dataset['test']:
    print(sample)

# for realworld, we have 'testmini', 'validation', 'test' splits
dataset = load_dataset("Wenliang04/HRScene", "realworld_combined")
for sample in dataset['test']:
    print(sample)

Data Format

WhiteBackground

id: int, image: PIL.JpegImagePlugin.JpegImageFile, question: str, answer: list[str]

{'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=448x448 at 0x7F01D88BF7A0>, 'id': 0, 'question': 'Is it daytime?', 'answer': ['no', 'no', 'no', 'no', 'no', 'no', 'no', 'no', 'no', 'no']}

ComplexGrid

id: str, image: PIL.JpegImagePlugin.JpegImageFile, caption: str, answer: str

{'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=1464x1524 at 0x7FB8634E6B70>, 'id': '0_0_0', 'caption': 'A nice living room has chairs and a love seat.', 'answer': 'row: 1, col: 1'}

RealWorld

id: int, image: PIL.Image.Image, question: str, answer: str

{'id': 0, 'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=5760x1200 at 0x7F4994CB75F0>, 'question': 'What is motion of the pedestrian wearing blue top on the left?\n(A) crossing the crosswalk\n(B) standing\n(C) jaywalking (illegally crossing not at pedestrian crossing)\n(D) walking on the sidewalk\n(E) The image does not feature the object', 'answer': 'None'}

🏆 Leaderboard 🏆

Leaderboard on the RealWorld Task on the Test Split (Show Top 5 Only)

We provide a simple pipeline for automatic model prediction and submission file geneartion! You can find pipeline from our Github under "🔮 Evaluations on HRScene for RealWorld Task" section

✅ Cite

@article{zhang2025hrscene,
    title={HRScene: How Far Are VLMs from Effective High-Resolution Image Understanding?},
    author={Zhang, Yusen and Zheng, Wenliang and Madasu, Aashrith and Shi, Peng and Kamoi, Ryo and Zhou, Hao and Zou, Zhuoyang and Zhao, Shu and Das, Sarkar Snigdha Sarathi and Gupta, Vipul and Lu, Xiaoxin and Zhang, Nan and Zhang, Ranran Haoran and Iyer, Avitej and Lou, Renze and Yin, Wenpeng and Zhang, Rui},
    journal={arXiv preprint},
    year={2025}
}