lingshu-medical-mllm/lingshu_training_data_medical_domain

<p align="center"> <img src="https://huggingface.co/lingshu-medical-mllm/Lingshu-I-8B/resolve/main/lingshulogo.png" width="200" /> </p> <p align="center"> <a href="https://alibaba-damo-academy.github.io/lingshu/" target="blank" rel="noopener">Website</a> &nbsp;&nbsp; <a href="https://huggingface.co/lingshu-medical-mllm/Lingshu-7B" target="blank" rel="noopener"> 🤖 7B Model</a> &nbsp;&nbsp; <a href="https://huggingface.co/lingshu-medical-mllm/Lingshu-I-8B" target="blank" rel="noopener"> 🤖 8B Model based on InternVL3</a> &nbsp;&nbsp; <a href="https://huggingface.co/lingshu-medical-mllm/Lingshu-32B" target="blank" rel="noopener"> 🤖 32B Model</a> &nbsp;&nbsp; <a href="https://github.com/alibaba-damo-academy/MedEvalKit" target="blank" rel="noopener"> MedEvalKit </a> &nbsp;&nbsp; <a href="https://arxiv.org/abs/2506.07044" target="blank" rel="noopener">Technical Report</a> &nbsp;&nbsp; <a href="https://github.com/alibaba-damo-academy/LingshuMCP" target="_blank" rel="noopener">Lingshu MCP</a> </p>

Lingshu Medical MLLM Training Data (Medical Domain)

This dataset contains the medical-domain training data used in the multi-stage training of the Lingshu Medical Multimodal Large Language Model (MLLM). General-domain data has been removed; only medical data is included.

The training framework follows a "shallow-to-deep" progression comprising three sequential stages:

1. 1.Medical Shallow Alignment — Establish effective alignment between diverse medical imaging modalities and their corresponding textual descriptions.
2. 2.Medical Deep Alignment — Comprehensively integrate medical knowledge into the MLLM with enriched, semantically complex medical image-text pairs.
3. 3.Medical Instruction Tuning — Refine the model's instruction-following capability across a broad spectrum of clinical tasks.

Please refer to our technical report for the detailed description of our training data.

Dataset Overview

Total size: ~3.3 TB

Files exceeding 20 GB are split into chunks on line boundaries. Each chunk is a valid JSONL file and can be processed independently. To reconstruct the full dataset for a stage, simply concatenate all parts in order.

Data Format

Each line is a JSON object with the following fields:

{
  "id": "alignment_1",
  "images": ["data:image/jpeg;base64,..."],
  "conversations": [
    {"role": "user", "content": "Write a description of the image\n<image>"},
    {"role": "assistant", "content": "The image shows..."}
  ],
  "modality": "CT Scan",
  "resolutions": [[width, height]]
}

Please note that the modality of some samples are predicted by VLM and may not be 100% correct.

Stage Details

Stage 1: Medical Shallow Alignment

• —Goal: Enable the model to encode medical images and generate corresponding descriptions accurately.
• —Training strategy: LLM frozen; only the vision encoder and projector are fine-tuned.
• —Data characteristics: Coarsely annotated medical image-caption pairs with relatively short and concise captions.

Stage 2: Medical Deep Alignment

• —Goal: Deepen the model's domain knowledge and achieve finer-grained vision-language alignment.
• —Training strategy: All model parameters (LLM, vision encoder, projector) are unfrozen for end-to-end fine-tuning.
• —Data characteristics: Significantly expanded modality diversity, linguistically complex and structurally complete captions, including synthetic image-caption pairs generated from medical image classification and segmentation tasks.

Stage 3: Medical Instruction Tuning

• —Goal: Refine the model's ability to comprehend and execute task-specific instructions across various medical use cases.
• —Training strategy: All parameters unlocked for large-scale, end-to-end optimization.
• —Data characteristics: Diverse instruction formats including image descriptions, question-answer pairs, multiple-choice questions, diagnosis queries, clinical report generation, anatomical structure localization, multi-image reasoning, multi-turn dialogues, and chain-of-thought reasoning.

Imaging Modalities Covered

The dataset covers a wide range of medical imaging modalities, including:

• —X-Ray / Chest X-Ray
• —CT Scan
• —MRI (Magnetic Resonance Imaging)
• —Ultrasound
• —Histopathology
• —Dermoscopy
• —Microscopy
• —OCT (Optical Coherence Tomography)
• —Fundus Photography
• —Endoscopy
• —And more

Downloading the Dataset

The dataset is hosted at `lingshu-medical-mllm/lingshu_training_data_medical_domain`.

Option 1: Using the datasets library

Load a specific stage by pattern-matching its chunked files:

from datasets import load_dataset

REPO_ID = "lingshu-medical-mllm/lingshu_training_data_medical_domain"

# Stage 1 — Medical Shallow Alignment
stage1 = load_dataset(
    REPO_ID,
    data_files="stage_1_release_data_encoded.jsonl",
    split="train",
)

# Stage 2 — Medical Deep Alignment
stage2 = load_dataset(
    REPO_ID,
    data_files="stage_2_release_data_encoded_part_*.jsonl",
    split="train",
)

# Stage 3 — Medical Instruction Tuning
stage3 = load_dataset(
    REPO_ID,
    data_files="stage_3_release_data_encoded_part_*.jsonl",
    split="train",
)

For very large stages (Stage 2 and Stage 3), use streaming mode to avoid materializing the full dataset on disk:

stage3_stream = load_dataset(
    REPO_ID,
    data_files="stage_3_release_data_encoded_part_*.jsonl",
    split="train",
    streaming=True,
)
for sample in stage3_stream:
    # process sample
    pass

Option 2: Using huggingface_hub (recommended for full downloads)

For downloading the raw JSONL files directly (e.g., to a local disk or NAS for training), use snapshot_download. This is efficient for multi-TB downloads and supports resumption:

from huggingface_hub import snapshot_download

REPO_ID = "lingshu-medical-mllm/lingshu_training_data_medical_domain"

# Download the entire dataset
local_dir = snapshot_download(
    repo_id=REPO_ID,
    repo_type="dataset",
    local_dir="./lingshu_data",
)

# Or download only a specific stage
local_dir = snapshot_download(
    repo_id=REPO_ID,
    repo_type="dataset",
    local_dir="./lingshu_data",
    allow_patterns=["stage_2_release_data_encoded_part_*.jsonl"],
)

To download a single chunk:

from huggingface_hub import hf_hub_download

path = hf_hub_download(
    repo_id="lingshu-medical-mllm/lingshu_training_data_medical_domain",
    filename="stage_3_release_data_encoded_part_000.jsonl",
    repo_type="dataset",
    local_dir="./lingshu_data",
)

Usage

Loading a single chunk

import json

data = []
with open("stage_1_release_data_encoded.jsonl", "r") as f:
    for line in f:
        data.append(json.loads(line))

Streaming through chunked files

import json
from pathlib import Path

def stream_stage(stage_pattern: str):
    """Stream all samples from a stage's chunked files."""
    for path in sorted(Path(".").glob(stage_pattern)):
        with open(path, "r") as f:
            for line in f:
                yield json.loads(line)

# Example: iterate through all Stage 3 data
for sample in stream_stage("stage_3_release_data_encoded_part_*.jsonl"):
    # process sample
    pass

Decoding images

import base64
from io import BytesIO
from PIL import Image

def decode_image(data_uri: str) -> Image.Image:
    """Decode a base64 data URI to a PIL Image."""
    base64_str = data_uri.split("base64,")[1]
    image_bytes = base64.b64decode(base64_str)
    return Image.open(BytesIO(image_bytes))

Privacy and Data Disclaimer

Notice: All data included in this dataset are sourced strictly from publicly available datasets and open academic repositories. The dataset contains no personally identifiable information (PII) or protected health information (PHI), and does not compromise or disclose the personal privacy of individual patients.

Citation

If you use this dataset, please cite the Lingshu paper:

@article{xu2025lingshu,
  title={Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning},
  author={Xu, Weiwen and Chan, Hou Pong and Li, Long and Aljunied, Mahani and Yuan, Ruifeng and Wang, Jianyu and Xiao, Chenghao and Chen, Guizhen and Liu, Chaoqun and Li, Zhaodonghui and others},
  journal={arXiv preprint arXiv:2506.07044},
  year={2025}
}

License

This dataset is released under the MIT License.