dnamodel/xraydar-multimodal

X-Raydar Multimodal Chest X-Ray Dataset

A multimodal dataset of 979 chest X-ray examinations, each with:

1. 1.Chest X-ray image (full-resolution PNG, anonymised)
2. 2.Consensus image-level labels (37 radiological findings, agreed by two expert radiologists with adjudication)
3. 3.Bounding box annotations on the image from each annotator independently (localising each finding)
4. 4.Original radiology report text
5. 5.Report span annotations (token-level labels across 45 categories)

This dataset combines resources from two annotation processes described in the papers below. The image-level consensus labels were agreed by two experienced radiologists with a third senior radiologist adjudicating disagreements. Individual bounding boxes from each annotator are provided separately (identified by the annotator field), allowing researchers to study inter-annotator agreement or merge boxes as needed. The report text annotations were produced independently by a separate team of radiologists.

Website: x-raydar.info X-ray classifier: dnamodel/xraydar-cv · Code Report classifier: dnamodel/xraydar-nlp · Code Full annotated reports (29,756): dnamodel/xraydar-reports

Dataset Structure

images/                     # 979 chest X-ray PNG images (full resolution)
  {xray_id}.png
annotations.jsonl           # All annotations in one file

Annotation Format

Each line in annotations.jsonl is a JSON object:

{
  "xray_id": "1858",
  "image_file": "images/1858.png",
  "consensus_labels": ["consolidation", "mediastinum_widened", "unfolded_aorta", "volume_loss"],
  "bounding_boxes": [
    {"label": "consolidation", "annotator": "705", "x_min": 1200, "y_min": 800, "x_max": 1500, "y_max": 1100},
    {"label": "mediastinum_widened", "annotator": "703", "x_min": 700, "y_min": 200, "x_max": 1000, "y_max": 600}
  ],
  "report_text": "There is consolidation in the left lower zone...",
  "report_tokens": ["There", "is", "consolidation", "in", "the", "..."],
  "report_iobe_tags": ["B", "I", "I", "I", "I", "..."],
  "report_spans": [
    {"label": "consolidation", "start": 0, "end": 8}
  ],
  "report_labels": ["consolidation", "volume_loss"]
}

Annotation Sources

Note: The image annotations and report annotations were produced independently. Image labels reflect what is visible in the image; report labels reflect what was written by the original reporting radiologist. These do not necessarily agree — a finding may be visible but unreported, or mentioned in the report but not visually apparent.

Image-Level Finding Categories (37 classes)

Usage

from huggingface_hub import snapshot_download
import json
from PIL import Image

# Download dataset
snapshot_download(
    repo_id="dnamodel/xraydar-multimodal",
    repo_type="dataset",
    local_dir="./xraydar-multimodal"
)

# Load annotations
with open("./xraydar-multimodal/annotations.jsonl") as f:
    data = [json.loads(line) for line in f]

# Load an image with its annotations
exam = data[0]
img = Image.open(f"./xraydar-multimodal/{exam['image_file']}")
print(f"Image: {img.size}")
print(f"Consensus labels: {exam['consensus_labels']}")
print(f"Bounding boxes: {len(exam['bounding_boxes'])}")
print(f"Report: {exam['report_text'][:100]}...")

Data Provenance

• —Images: Frontal chest X-rays from three UK NHS hospital networks (2006–2019), extracted from DICOM as anonymised PNGs at native resolution.
• —Image annotations: Produced using the AnnotateX platform. Two experienced radiologists independently annotated each image with finding labels and bounding boxes. Image-level consensus labels were agreed through adjudication with a third senior radiologist. Individual bounding boxes from each annotator are preserved with their annotator IDs.
• —Report text: Historical free-text radiology reports associated with each examination.
• —Report annotations: 29,756 reports were manually annotated by ten radiologists using the AnnotateX platform, with span-level labels across 45 categories.

Citation

If you use this dataset, please cite:

@article{cid2024development,
  title={Development and validation of open-source deep neural networks for comprehensive chest
         x-ray reading: a retrospective, multicentre study},
  author={Cid, Yan Digilov and Macpherson, Matt and others},
  journal={The Lancet Digital Health},
  volume={6}, number={1}, pages={e44--e57},
  year={2024}, publisher={Elsevier},
  doi={10.1016/S2589-7500(23)00218-2}
}

@inproceedings{zhu2024multitask,
  title={A Multi-Task Transformer Model for Fine-grained Labelling of Chest {X}-Ray Reports},
  author={Zhu, Yuanyi and Liakata, Maria and Montana, Giovanni},
  booktitle={Proceedings of the 2024 Joint International Conference on Computational Linguistics,
             Language Resources and Evaluation (LREC-COLING 2024)},
  pages={862--875},
  year={2024},
  publisher={ELRA and ICCL}
}

License

For academic research and non-commercial evaluation only. See x-raydar.info for terms and conditions.

Contact

Giovanni Montana — g.montana@warwick.ac.uk