CoolFace
Modelpublic

dmusingu/diffvqa-qwen3-0.6b-multiobjective-regsteps

sourceHugging Facemitupdated 2mo agoView on Hugging Face
0likes
Model Card

Diff-VQA — Qwen3-0.6B visual-prefix decoder (multiobjective-regsteps encoder)

Online-trained Difference Visual Question Answering head for chest X-rays (MIMIC-CXR / Medical-Diff-VQA). A frozen ViT-L/14 vision encoder produces patch tokens for a current + reference image pair; a Qwen3-0.6B decoder, conditioned on those tokens as a visual prefix, generates the answer describing what changed.

This is a slimmed inference checkpoint: decoder weights + the frozen vision encoder weights + architecture metadata (optimizer/scheduler state stripped).

Contents (*_best.pt, torch.load(..., weights_only=False))

keydescription
state_dictDiffVQAQwen3Head weights (Qwen3-0.6B LLM + vis_proj + frame_emb)
encoder_statefrozen ViT-L/14 vision encoder (objective: multiobjective_caption+regioncontrastive)
decoderQwen/Qwen3-0.6B
tokenizer_nameQwen/Qwen3-0.6B
vis_dim1024
encoder_namepretrain-multiobjective-regsteps
epoch / step / best_valtraining position + best validation loss

Training

  • —Data: Medical-Diff-VQA difference questions over MIMIC-CXR image pairs (~100k train / ~16k val / ~16k test).
  • —Objective: next-token cross-entropy over the answer span (question tokens masked).
  • —Vision encoder is frozen; only the decoder (LLM + visual adapter) is trained.

Notes

The bundled encoder_state is the exact frozen encoder this decoder was trained against — load both together so the visual features match the space vis_proj expects (a mismatched encoder produces degenerate output).