Yale-BIDS-Chen/M3LLM-data

M3LLM-PMC Training Data

This dataset contains the training data for M3LLM (Medical Multimodal Large Language Model), comprising ~238K high-quality synthetic medical instruction-following samples.

Dataset Description

The data is generated from PubMed Central (PMC) medical literature through a comprehensive 5-stage synthetic data pipeline, covering six diverse medical visual question answering tasks.

Dataset Statistics

Data Format

Each JSONL file contains one JSON object per line with the following structure:

{
  "image": "path/to/image.jpg",
  "caption": "Original image caption",
  "qa_pairs": [
    {
      "question": "Medical question about the image",
      "answer": "Detailed medical answer",
      "context": "Additional context (task-dependent)",
      "improved context": "Refined context without answer leakage"
    }
  ]
}

Usage

Loading with Datasets Library

from datasets import load_dataset

# Load entire dataset
dataset = load_dataset("KerwinFu/M3LLM-PMC")

# Load specific task
puretext_data = load_dataset("KerwinFu/M3LLM-PMC", data_files="puretext.jsonl")

Manual Download

# Download all files
git clone https://huggingface.co/datasets/KerwinFu/M3LLM-PMC

# Or download specific files
wget https://huggingface.co/datasets/KerwinFu/M3LLM-PMC/resolve/main/puretext.jsonl

Data Generation Pipeline

The data is synthesized through a 5-stage pipeline:

1. 1.Stage 1-3: Preprocessing
2. 2.Inline text summarization
3. 3.Medical knowledge extraction
4. 4.Visual perception enhancement

1. 1.Stage 4: Task-specific QA generation
2. 2.Six specialized scripts for different medical VQA tasks
3. 3.Uses Qwen2.5-32B-Instruct for high-quality generation

1. 1.Stage 5: Context refinement
2. 2.Removes answer-revealing information
3. 3.Ensures data quality and prevents leakage

For detailed pipeline documentation, see the M3LLM repository.

Model Training

This dataset is used to finetune InternVL3-8B to create M3LLM.

Training configurations:

• —LoRA finetuning: Rank 16, frozen vision backbone
• —Full model finetuning: Trainable LLM + MLP, frozen vision backbone

See training documentation for details.

Citation

If you use this dataset, please cite:

@article{m3llm2024,
  title={M3LLM: Medical Multimodal Large Language Model},
  author={[Your Name and Collaborators]},
  journal={arXiv preprint arXiv:XXXX.XXXXX},
  year={2024}
}

License

This dataset is released under the MIT License. Please also cite the original PMC sources when using this data.

Acknowledgments

• —PMC Open Access Subset for source medical literature
• —InternVL for the base model
• —Qwen2.5 for synthetic data generation

Contact

For questions or issues, please open an issue on the M3LLM GitHub repository.