Datasetpublic

Silversorrow/SpineBench

Dataset Card for SpineBench Benchmark Details Paper Information Benchmark Examples Benchmark Distribution Data Format Data Source Human Evaluation of MLLMs Reasoning Performance Citation Benchmark Details SpineBench is a comprehensive Visual Question Answering (VQA) benchmark designed for fine-grained analysis and evaluation of LVLM in the spinal domain. SpineBench comprises 64,878 QA pairs from 40,263 spine images, covering 11 spinal diseases through two… See the full description on the dataset page: https://huggingface.co/datasets/Silversorrow/SpineBench.

sourceHugging Faceapache-2.0updated 10mo agoView on Hugging Face

0likes9.9kdownloads

Dataset Card

Dataset Card for SpineBench

—Benchmark Details
—Paper Information
—Benchmark Examples
—Benchmark Distribution)
—Data Format
—Data Source
—Human Evaluation of MLLMs Reasoning Performance
—Citation

Benchmark Details

SpineBench is a comprehensive Visual Question Answering (VQA) benchmark designed for fine-grained analysis and evaluation of LVLM in the spinal domain. SpineBench comprises 64,878 QA pairs from 40,263 spine images, covering 11 spinal diseases through two critical clinical tasks: spinal disease diagnosis and spinal lesion localization.

Paper Information

—Project: https://zhangchenghanyu.github.io/SpineBench.github.io/
—Appendix: https://zhangchenghanyu.github.io/SpineBench.github.io/#Appendix

Benchmark Example

Examples of SpineBench cases:

<details> <summary>🔍 Click to expand/collapse more examples</summary>

Examples of more SpineBench cases:

1.Correct Disease Diagnose:

1.Healthy Patient Diagnose:

1.Disease Misdignose:

1.Healthy Patient Misdiagnosis:

</details>

Benchmark Distribution

The evaluation set is derived from SpineBench.

—SpineBench: 40,263 spine images, each annotated with one of the 11 disease types. Among these, 24,615 images also have detailed lesion localization annotations across five lumbar spinal segments (L1/L2, L2/L3, L3/L4, L4/L5, and L5/S1).
—SpineBench Evaluation Set: 1,000 cases for the spinal disease diagnosis task and 1,128 cases for the spinal lesion localization task.

You can download this dataset by the following command (make sure that you have installed Huggingface Datasets):

python

from datasets import load_dataset

dataset = load_dataset("Silversorrow/SpineBench")

Disease distribution of SpineBench:

Disease distribution of the evaluation set in SpineBench:

Data Format

The dataset is provided in json format and contains the following attributes:

json

{
    "question_id" [string] Case ID, e.g., "1",
    "task": [string] Evaluation task of the case (spinal disease diagnosis/spinal lesion localization),
    "question": [string] The question text,
    "image": [string] A file path pointing to the associated image,
    "options": [list] Choice options for multiple-choice problems. For spinal disease diagnosis problems, there must have a 'healthy' choice; For spinal lesion localization problems, there are five lumbar spinal segments (L1/L2, L2/L3, L3/L4, L4/L5, and L5/S1),
    "answer": [string] The correct answer for the problem,
    "metadata" (if the case have): {
        "gender": [string] Patient gender (male/female),
        "age": [string] Patient age,
		}
}

Data Source

The SpineBench dataset is derived from four newly collected datasets: BUU Spine Dataset, CSXA, RSNA, and VinDr-SpineXR. All these source datasets have been preprocessed and labeled for evaluation purposes.

—BUU Spine Dataset:https://services.informatics.buu.ac.th/spine/
—CSXA:https://www.scidb.cn/en/detail?dataSetId=8e3b3d5e60a348ba961e19d48b881c90
—RSNA:https://www.kaggle.com/competitions/rsna-2024-lumbar-spine-degenerative-classification
—VinDr-SpineXR:https://physionet.org/content/vindr-spinexr/1.0.0/

Human Evaluation of MLLMs Reasoning Performance

we invite collaborating clinical physicians to evaluate model reasoning processes in spinal disease diagnosis and lesion localization tasks across four dimensions: 1) Clinical Plausibility: Whether the model’s reasoning aligns with clinical logic and remains consistent with established medical knowledge, diagnostic principles, and anatomical understanding relevant to spinal pathology. 2) Identification and Utilization of Key Visual Features: Whether the reasoning accurately identifies and prioritizes relevant visual evidence in the image to support diagnostic conclusions. 3) Depth of Pathological Understanding: Whether the reasoning reflects superficial pattern matching or demonstrates deeper comprehension of the specific pathophysiology presented. 4) Quality of Spatial Reasoning (spinal lesion localization task only): Evaluates the reasoning process’s ability to demonstrate an understanding of 3D spatial relationships within the spine.

Citation

If you use the SpineBench dataset in your work, please kindly cite the paper using this BibTeX:

@inproceedings{zhang2025spinebench, title={SpineBench: Benchmarking Multimodal LLMs for Spinal Pathology Analysis}, author={Zhang, Chenghanyu and Li, Zekun and Li, Peipei and Cui, Xing and Xia, Shuhan and Yan, Weixiang and Zhang, Yiqiao and Zhuang, Qianyu}, booktitle={Proceedings of the 33rd ACM International Conference on Multimedia}, pages={12729--12736}, year={2025} }