DolphinAI/u2-bench

1---2license: cc-by-nc-nd-4.03task_categories:4- question-answering5language:6- en7tags:8- medical9size_categories:10- 1K<n<10K11---12 13# U2-BENCH: Ultrasound Understanding Benchmark14 15**U2-BENCH** is the **first large-scale benchmark for evaluating Large Vision-Language Models (LVLMs) on ultrasound imaging understanding**. It provides a diverse, multi-task dataset curated from **40 licensed sources**, covering **15 anatomical regions** and **8 clinically inspired tasks** across classification, detection, regression, and text generation.16 17### Check the 🌟Leaderboard🌟here: https://dolphin-sound.github.io/u2-bench/18 19### Evaluation code released! 20Two options:211. developed from VLMEvalKit: https://github.com/dolphin-sound/u2-bench-evalkit222. developed by our intern Yalun: https://github.com/gurenolun/Dolphin-ai-bench23 24---25 26## 📂 Dataset Structure27 28The dataset is organized into **8 folders**, each corresponding to one benchmark task:29 30- `caption_generation/`31- `clinical_value_estimation/`32- `disease_diagnosis/`33- `keypoint_detection/`34- `lesion_localisation/`35- `organ_detection/`36- `report_generation/`37- `view_recognition_and_assessment/`38 39Each folder contains `.tsv` files with task-specific annotations. A shared file, [`an_explanation_of_the_columns.tsv`](./an_explanation_of_the_columns.tsv), maps each column to its meaning.40 41---42 43## 📄 Data Format44 45The dataset is provided as `.tsv` files, where:46 47- `img_data` contains a **base64-encoded image** (typically a 2D frame from an ultrasound video).48- Each row corresponds to a **single sample**.49- Columns include task-specific fields such as:50  - `dataset_name`, `anatomy_location`, `classification_task`51  - `caption`, `report`, `class_label`, `measurement`, `gt_bbox`, `keypoints`, etc.52 53A full explanation is provided in [`an_explanation_of_the_columns.tsv`](./an_explanation_of_the_columns.tsv).54 55---56 57## 🔬 Tasks58 59U2-BENCH includes 8 core tasks:60 61| Capability     | Task Name                    | Description                                     |62|----------------|------------------------------|-------------------------------------------------|63| Classification | Disease Diagnosis (DD)       | Predict clinical diagnosis from ultrasound      |64| Classification | View Recognition (VRA)       | Classify standard views in sonography           |65| Detection      | Lesion Localization (LL)     | Locate lesions with spatial classification      |66| Detection      | Organ Detection (OD)         | Identify presence of anatomical structures      |67| Detection      | Keypoint Detection (KD)      | Predict anatomical landmarks (e.g. biometry)    |68| Regression     | Clinical Value Estimation    | Estimate scalar metrics (e.g., fat %, EF)       |69| Generation     | Report Generation            | Produce structured clinical ultrasound reports  |70| Generation     | Caption Generation           | Generate brief anatomical image descriptions    |71 72---73 74## 📊 Dataset Statistics75 76- **Total samples**: 7,24177- **Anatomies**: 15 (e.g., thyroid, fetus, liver, breast, heart, lung)78- **Application scenarios**: 50 across tasks79- **Multi-task support**: Some samples contain multiple labels (e.g., classification + regression)80 81---82 83## 🛡️ Ethics, License & Use84 85- The dataset is distributed under the **Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0)** license.86- For **non-commercial research and evaluation only**.87- Data is derived from **licensed and publicly available ultrasound datasets**.88- All images are de-identified, and annotations were manually validated.89- **Do not use** this dataset for diagnostic or clinical deployment without regulatory approval.90 91---92 93## 📦 Loading from Hugging Face94 95You can load the dataset using 🤗 Datasets:96 97```python98from datasets import load_dataset99 100dataset = load_dataset("DolphinAI/u2-bench", split="train")101```102 103---104 105## 📚 Citation106 107If you use this benchmark in your research, please cite:108 109```bibtex110@article{le2025u2bench,111  title={U2-BENCH: Benchmarking Large Vision-Language Models on Ultrasound Understanding},112  author={Le, Anjie and Liu, Henan and others},113  journal={Under Review},114  year={2025}115}116```117 118---119 120## 🔧 Contributions121 122We welcome community contributions and evaluation scripts.123Please open a pull request or contact Dolphin AI for collaboration.124