erow/Medical-ROIs-K2.6

Medical-ROIs-K2.6

Medical visual grounding SFT data: for each clinical VQA sample, a teacher model proposes answer-supporting ROIs (regions of interest) as 2D bounding boxes.

Teacher: `moonshotai/Kimi-K2.6`. Upstream images & QA: `MBZUAI/medix-rl-data`.

How the data is generated

MBZUAI/medix-rl-data (train)
        │
        │  problem / solution / image / source / id
        ▼
  Teacher: moonshotai/Kimi-K2.6
  (vision + text; given question + gold answer)
        │
        │  ROI XML: <region><title>…</title><bbox>[x1,y1,x2,y2]</bbox></region>
        ▼
  JSONL (sft.jsonl)  →  stratified split
        │                 ├── sft_train.jsonl  (25,134)
        │                 └── sft_val.jsonl    (2,000)
        ▼
  prepare_hf: join JSONL text with medix-rl-data images
        │
        ▼
  Hugging Face DatasetDict  (this repo)

1. Source VQA pool

Rows come from `MBZUAI/medix-rl-data` (train, 51,335 examples) with four upstream sources:

Each row provides id, image, problem (question), solution (gold answer), and source.

2. Teacher ROI annotation

A multimodal teacher (moonshotai/Kimi-K2.6) is prompted with the image, clinical question, and gold answer, and asked to box the visual evidence that supports the answer.

• —Output format (one or more blocks):

<region><title>caption</title><bbox>[x1,y1,x2,y2]</bbox></region>

• —BBox convention: integers in [0, 1000], top-left origin; require x1 < x2, y1 < y2.
• —Successful annotations are written to JSONL (sft.jsonl); failures are logged for repair/retry.
• —This release keeps the successfully annotated subset (27,134 rows, ≈53% of the upstream train pool).

3. Train / validation split

From sft.jsonl, a source-stratified random split (seed=42) produces:

4. Hugging Face packaging

medix_seeing_sft.prepare_hf joins each JSONL row with the matching image(s) from MBZUAI/medix-rl-data by id, and materializes a DatasetDict with multimodal chat fields suitable for SFT loaders.

Splits in this repo

Source mix (approx.)

Schema

Example assistant target

<region><title>Right kidney</title><bbox>[130,380,470,720]</bbox></region>
<region><title>Left kidney</title><bbox>[570,420,840,750]</bbox></region>

Load

from datasets import load_dataset

ds = load_dataset("erow/Medical-ROIs-K2.6")
print(ds["train"][0]["problem"])
print(ds["train"][0]["response"])
ds["train"][0]["images"][0]  # PIL.Image

Or from a local save_to_disk folder:

from datasets import load_from_disk
ds = load_from_disk("/path/to/hf_seeing_sft")

Intended use

Supervised fine-tuning / distillation of medical VLMs for visual grounding: predicting ROIs that evidence the answer to a clinical question (complementary to answer-only VQA).

Limitations

• —ROIs are teacher-generated, not human-verified for every sample; some boxes may be loose, panel-misaligned, or weakly linked to the QA.
• —Coverage is a subset of medix-rl-data (annotation failures / rate limits excluded).
• —Not a clinical decision-support product; for research use only.

Citation / provenance

• —Upstream pool: MBZUAI/medix-rl-data and its constituent VQA datasets (PMC-VQA, PathVQA, SLAKE, VQA-RAD).
• —Teacher: Moonshot AI Kimi K2.6 (via OpenAI-compatible inference).
• —Pipeline: MediX seeing-SFT (medix_seeing_sft generate → split → prepare_hf).