HoqueMahmudul/llava-onevision-7b-qlora-radiology-image-caption

LLaVA-OneVision 7B QLoRA — Radiology Image Captioning

A parameter-efficient fine-tune of LLaVA-OneVision (7B parameters) for medical (radiology) image captioning, adapted with QLoRA. Given a radiology image, it generates a short free-text caption describing the image.

Part of the CS_Morgan Lab submission to ImageCLEFmedical Caption 2026.

Project: this model is one of seven in the Radiology Image Captioning collection — LLaVA-OneVision-Qwen2 at 0.5B, 7B and 72B, each adapted with LoRA, QLoRA or QDoRA.

Code: medical-vlm-explainability — training, caption generation, and the attention-based explainability pipeline.

What this is (read first)

This repo contains a PEFT adapter, not a standalone model. The adapter is a small set of low-rank weight updates; the ~~16 GB of original base-model weights are not here. At load time you fetch two things:

1. 1.the base model `llava-hf/llava-onevision-qwen2-7b-ov-hf`, and
2. 2.this adapter, applied on top.

Nothing in this repo runs without that exact base model.

Files in this repo

Note there is no `config.json` and no `preprocessor_config.json` here, since those belong to the base model. This is why the examples below load the processor from the base — see Troubleshooting.

Installation

pip install "transformers>=4.45" "peft>=0.9" accelerate safetensors pillow
# 4-bit loading (optional, recommended for the 7B and 72B):
pip install bitsandbytes

Versions used at training time:

transformers==5.3.0
peft==0.18.1
bitsandbytes==0.49.2
torch==2.11.0+cu128

Quick start — caption one image

import torch
from PIL import Image
from transformers import AutoProcessor, LlavaOnevisionForConditionalGeneration
from peft import PeftModel

BASE = "llava-hf/llava-onevision-qwen2-7b-ov-hf"
ADAPTER = "HoqueMahmudul/llava-onevision-7b-qlora-radiology-image-caption"

# 1. base model
model = LlavaOnevisionForConditionalGeneration.from_pretrained(
    BASE, torch_dtype=torch.bfloat16, device_map="auto"
)
# 2. this adapter on top
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

# 3. processor from the BASE (not from this repo -- see Troubleshooting)
processor = AutoProcessor.from_pretrained(BASE)

# 4. build the prompt with the SAME text used in training
image = Image.open("your_image.jpg").convert("RGB")
conversation = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "Describe this medical image."},
        ],
    }
]
prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
inputs = processor(images=image, text=prompt, return_tensors="pt").to(model.device)

# 5. greedy decoding, as used for the ImageCLEF submission
with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=256, do_sample=False)

caption = processor.decode(
    out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True
)
print(caption.strip())

Use the prompt `"Describe this medical image."` verbatim. The adapter was trained with only that instruction; other phrasings are out of distribution and degrade output quality.

The rendered prompt should look exactly like this:

<|im_start|>user <image>
Describe this medical image.<|im_end|><|im_start|>assistant

Lower memory: load the base in 4-bit

This adapter was trained against a 4-bit NF4-quantized base, so loading the base in 4-bit reproduces the training-time setup and cuts VRAM substantially:

from transformers import BitsAndBytesConfig

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)
model = LlavaOnevisionForConditionalGeneration.from_pretrained(
    BASE, quantization_config=bnb, device_map="auto"
)
model = PeftModel.from_pretrained(model, ADAPTER)

Captioning many images

def caption(image_path):
    image = Image.open(image_path).convert("RGB")
    inputs = processor(images=image, text=prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
    return processor.decode(
        out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True
    ).strip()

for path in ["a.jpg", "b.jpg", "c.jpg"]:
    print(path, "->", caption(path))

The processor supports batching, but LLaVA-OneVision expands each image into a variable number of visual tokens depending on resolution, so padded batches need processor.tokenizer.padding_side = "left" for correct generation. Looping one image at a time is simplest and is what the original evaluation did.

Hardware requirements

Measured on 500 validation images, batch size 1, greedy decoding with max_new_tokens=256 (the setting used for the published captions), on the GPU named below. Figures are torch.cuda.max_memory_allocated() on a single device holding a complete model — the model is replicated, not sharded.

Comparing these numbers across variants requires care. Scales were measured on different GPUs (0.5B on A100, 7B and 72B on H200), so latency is not comparable across scales. Within a scale it is comparable only when the base loading also matches: QLoRA vs QDoRA is a fair comparison (same GPU, both 4-bit), but LoRA vs QLoRA is not — those differ in base quantization as well as PEFT method, so the gap conflates the two.

Why peak memory exceeds the weight size. LLaVA-OneVision uses anyres tiling: a large image expands into thousands of visual tokens (1024x768 -> ~5,100 tokens; 1920x1080 -> ~11,700), and those activations dominate. Quantizing the base to 4-bit shrinks weights but not activations, so 4-bit does not reduce peak inference memory proportionally — budget from the measured figure, not from parameter count.

Peak scales with input resolution and max_new_tokens; smaller images need much less. The adapter itself adds only tens to hundreds of MB.

Serving several adapters on one base

Loading several adapters onto one resident base and switching with set_adapter() was tested against separately-loaded references: 20 validation images, 4-bit NF4 base, greedy decoding, `max_new_tokens=256`, across every ordered pair in this series.

Result: adding a second adapter changed nothing — 0/20 captions differed. Hot-swapping is safe for these adapters, in either load order.

One caveat, separate from hot-swapping. Loading an adapter under a non-default adapter_name produced different captions than loading it with the default name — 9/20 images on the same base with identical inputs and greedy decoding. The outputs are not worse, but they are not bit-reproducible across the two loading forms. If you need to reproduce a previous run exactly, load the adapter the same way you did originally. (peft 0.18.1, transformers 5.3.0, torch 2.11.0+cu128, bitsandbytes 0.49.2)

Continuing fine-tuning from this adapter

You can keep training these weights on your own data. Two options:

Option A — continue training this adapter

model = LlavaOnevisionForConditionalGeneration.from_pretrained(
    BASE, torch_dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(model, ADAPTER, is_trainable=True)  # <-- REQUIRED
model.print_trainable_parameters()   # must report a non-zero trainable count

`is_trainable=True` is not optional. Without it, PEFT loads the adapter in inference mode with requires_grad=False; training then runs to completion without ever updating the adapter. Always confirm print_trainable_parameters() reports non-zero.

Things to know:

• —This is a warm start, not an exact resume. Optimizer and scheduler state are deliberately not published, so you begin with a fresh optimizer. Use a learning rate lower than the original 1e-4 to avoid washing out the learned weights.
• —The adapter's shape is fixed by adapter_config.json (r=16, alpha=32, and the target-module list below). To change rank or target modules you must train a new adapter — see Option B.

Option B — train a fresh adapter, using this one only as a reference

from peft import LoraConfig, get_peft_model

cfg = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    use_dora=False,
    target_modules=[
        "q_proj", "k_proj", "v_proj",          # LLM attention
        "gate_proj", "up_proj", "down_proj",   # LLM MLP
        "linear_1", "linear_2",                # multimodal projector
    ],
)
model = get_peft_model(base_model, cfg)

The SigLIP vision encoder, the language-model head, and the token embeddings were kept frozen during the original training.

Data format and collation

Each training example is one image plus one target caption, rendered through the same chat template. Sketch of a collator:

def collate(batch):
    prompts, images = [], []
    for ex in batch:                     # ex = {"image": PIL.Image, "caption": str}
        conv = [
            {"role": "user", "content": [
                {"type": "image"},
                {"type": "text", "text": "Describe this medical image."},
            ]},
            {"role": "assistant", "content": [
                {"type": "text", "text": ex["caption"]},
            ]},
        ]
        prompts.append(processor.apply_chat_template(conv))
        images.append(ex["image"])

    out = processor(images=images, text=prompts, return_tensors="pt", padding=True)
    labels = out["input_ids"].clone()
    labels[labels == processor.tokenizer.pad_token_id] = -100
    out["labels"] = labels               # mask prompt tokens too, to train on the answer only
    return out

The inference examples above were run and verified. The training snippets are templates — adapt them to your dataset, and mask the prompt tokens in labels if you want loss on the caption only.

To reproduce the original setup, mirror the recipe below: effective batch size 16, bf16, adamw_torch, warmup ratio 0.03, weight decay 0.01, validation loss evaluated every 500 steps, early stopping after four evaluations without improvement, best checkpoint restored by validation loss.

Merging the adapter into the base (optional)

For one-line loading later, you can bake the adapter into a full model:

model = LlavaOnevisionForConditionalGeneration.from_pretrained(
    BASE, torch_dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(model, ADAPTER)
merged = model.merge_and_unload()
merged.save_pretrained("./merged-model")
processor.save_pretrained("./merged-model")

Caveats: the result is the full model size (~16 GB), not the adapter size. Because this adapter was trained against a 4-bit base, merging into a bf16 base introduces a small numerical mismatch; this is standard practice and usually harmless, but validate on your data. Merging into a 4-bit-quantized base is not supported — load the base in bf16 to merge.

Troubleshooting

`OSError` / `LocalEntryNotFoundError` mentioning `huggingface.co` when loading the processor. You called AutoProcessor.from_pretrained(ADAPTER). This repo has no config.json/preprocessor_config.json, so the processor class can't be resolved from it; online this fails over silently, offline it errors. Load the processor from the base instead: AutoProcessor.from_pretrained(BASE). It is equivalent — the base processor's chat template is byte-identical to chat_template.jinja here, and its tokenizer encodes the training prompt to the same token ids.

Training runs but the adapter never changes. You omitted is_trainable=True in PeftModel.from_pretrained. See Option A.

`TypeError: can only concatenate str (not "list") to str` when applying the chat template. You used the tokenizer's template (text-only) instead of the processor's (multimodal). Call processor.apply_chat_template(...).

DoRA keys missing / unexpected, or magnitude vectors ignored. Upgrade to peft >= 0.9.

Out of memory. Load the base in 4-bit (see above), reduce image resolution, or use a smaller variant in this series.

Captions are generic or off-topic. Check that your prompt is exactly "Describe this medical image." and that the adapter actually loaded (model.peft_config should be populated).

Training data

The updated version of ROCOv2 (Radiology Objects in COntext v2), the dataset adopted by ImageCLEFmedical 2026 — radiology figures from the PubMed Central Open Access subset, each paired with a caption and UMLS concept metadata, spanning chest X-ray, CT, MRI, ultrasound, echocardiography, angiography, mammography, retinal fundus, dental panoramic, microscopy and pathology.

The partition is the one fixed by the task organizers, split by image-ID prefix rather than randomly: a captioned development pool of 116,604 images divided into 97,364 training / 19,240 validation, plus a separate 15,249-image test set released without reference captions.

The dataset is not redistributed here. Obtain it from ImageCLEF / ROCOv2 directly, and credit both as the source of the training data.

Fine-tuning recipe (as trained)

Identical across all variants in this series except the learning rate.

• —LoRA rank r=16, alpha=32 (ratio 2.0), dropout 0.05
• —Target modules: q_proj, k_proj, v_proj, gate_proj, up_proj, down_proj, linear_1, linear_2 (LLM attention + MLP, and the multimodal projector — the projector is included because it is where vision-to-language alignment is formed)
• —Quantization: 4-bit NormalFloat (NF4), double quantization, compute dtype bfloat16
• —DoRA: not used
• —Precision bf16, optimizer adamw_torch, warmup ratio 0.03, weight decay 0.01
• —Learning rate: 1e-4
• —Effective batch size 16 (per-device 1 x grad-accum 2 x 8 GPUs)
• —Early stopping on validation loss; best checkpoint restored
• —Inference: greedy decoding, max_new_tokens=256 (verified from the published caption outputs; an earlier revision of this card said 128)
• —Hardware: one node of 8x A100 40 GB, bfloat16, gradient checkpointing, PyTorch DistributedDataParallel via torchrun

Evaluation

This system was evaluated by the ImageCLEFmedical Caption 2026 organizers on the official held-out test set of 15,249 radiology images (references not publicly released). It was one of four systems submitted by the CS_Morgan Lab (0.5B and 7B, each with QLoRA and QDoRA), one run per system, using greedy decoding as described above.

The caption-prediction subtask was scored by six automated metrics, which the organizers group into two aspects and average into a single overall score:

Relevance aspect

Factuality aspect

Relevance is the mean of the four relevance metrics, factuality the mean of the two factuality metrics, and the overall score the mean of the two aspects.

The same system family was additionally evaluated in the ImageCLEFmedical 2026 explainability subtask, where a radiologist scored submissions on a five-point scale across nine criteria: readability, accuracy, level of detail, caption focus, visualization consistency, comprehensiveness, visualization focus, methodology, and clinician's favourite. That subtask scored each team's per-image submission rather than each individual system.

Score values are not reproduced on this card.

Intended use

Research and educational use in medical image understanding, and as a starting point for further fine-tuning.

Limitations and caveats

• —NOT a medical device. Outputs can be wrong or hallucinated. Do not use to inform patient care.
• —This is an adapter and does not run without the exact base model above.
• —Trained only on the ImageCLEFmedical 2026 / ROCOv2 distribution; performance on other distributions is unknown.

Citation

If you use this model, please cite:

@inproceedings{hoque2026csmorgan,
  title     = {Model-Intrinsic Attention as Explanation for Radiology Image Captioning},
  author    = {Hoque, Mahmudul and Chowdhury, Raisa Nusrat and
               Oluwafemi, Ejiga Peter Ojonugwa and Islam, Okib Ul and
               Hoque, Rahmanul and Rahman, Md Mahmudur},
  booktitle = {CLEF 2026 Working Notes},
  series    = {CEUR Workshop Proceedings},
  publisher = {CEUR-WS.org},
  address   = {Jena, Germany},
  year      = {2026}
}

Please also credit the dataset (ROCOv2) and the base model (LLaVA-OneVision).

Attribution

Mahmudul Hoque, Morgan State University — <mahoq1@morgan.edu> CS_Morgan Lab, Computer Science Department, Morgan State University.