mahdisetti/xray-qwen-lora

Chest X-ray Report Generation — Qwen2.5-VL 7B + RAG

Fine-tuned vision-language model that reads a chest X-ray image and generates a short radiology-style findings paragraph. Built by a first-year engineering student using only a free Colab T4 GPU.

⚠️ Research prototype only — not for medical diagnosis or clinical use.

What it does

Upload a chest X-ray → the system retrieves the most visually similar historical case (CLIP + FAISS) → injects it as context → fine-tuned Qwen2.5-VL 7B generates a findings paragraph.

Example output:

"Cardiomegaly noted; no new consolidations identified in lungs bilaterally compared to prior studies."

How it works

1. 1.Chest X-ray image uploaded via Gradio interface
2. 2.CLIP (ViT-B/32) encodes the image into an embedding vector
3. 3.FAISS searches a pre-built index of 62 similar X-ray cases
4. 4.Most visually similar historical report injected as context (RAG)
5. 5.Fine-tuned Qwen2.5-VL 7B generates the findings paragraph

Model details

Quick start

from unsloth import FastVisionModel
from PIL import Image
import torch

model, tokenizer = FastVisionModel.from_pretrained(
    model_name="mahdisetti/xray-qwen-lora",
    load_in_4bit=True,
)
FastVisionModel.for_inference(model)

image = Image.open("your_xray.jpg").convert("RGB")
image.thumbnail((512, 512))

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image},
            {"type": "text", "text": (
                "Write only one short radiology findings paragraph "
                "under 50 words. Mention the main visible abnormality "
                "and its anatomical location."
            )},
        ],
    }
]

input_text = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True
)
inputs = tokenizer(
    image, input_text,
    add_special_tokens=False,
    return_tensors="pt"
).to("cuda")

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=70,
        do_sample=False,
        repetition_penalty=1.3,
        no_repeat_ngram_size=5,
    )

new_tokens = outputs[0][inputs["input_ids"].shape[-1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

RAG files

FAISS index and report pickle stored at mahdisetti/xray-rag-files

Known limitations

• —Trained on only 50 examples — outputs should not be trusted clinically
• —Occasional hallucinations on ambiguous scans
• —Sensitive to prompt wording
• —May mislocalize findings (e.g. left vs right)
• —No formal evaluation metrics computed (BLEU/ROUGE planned)

Dataset used

X-iZhang/CheXpert-plus-RRG

Author

Mahdi Setti — first-year engineering student Built as a personal learning project to explore fine-tuning, multimodal models, and RAG in a real-world domain.