rockyaaos/medgemma-1.5-4b-preventive-qlora

MedGemma 1.5 4B · Preventive-Health QLoRA adapter (v1)

Research / education only. Synthetic training data only. NOT a diagnostic tool, not clinical decision support, not a medical device. Do not use with real patient data or PHI.

A LoRA/QLoRA adapter (≈30M params, ~57 MB) trained on top of google/medgemma-1.5-4b-it to produce a structured, non-diagnostic 7-part assessment of a synthetic longitudinal patient record. This is not a standalone model — you need separate access to the gated base model.

Part of a controlled study: github.com/rockyzl/preventive-health-model-lab. This is the medical arm; the control (same recipe, non-medical base) is `gemma-3-4b-preventive-qlora`.

Usage

import torch
from transformers import AutoTokenizer, BitsAndBytesConfig, Gemma3ForConditionalGeneration
from peft import PeftModel

base = "google/medgemma-1.5-4b-it"          # accept its license on HF first
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16)
tok = AutoTokenizer.from_pretrained(base)
model = Gemma3ForConditionalGeneration.from_pretrained(base, quantization_config=bnb, device_map="cuda")
model = PeftModel.from_pretrained(model, "rockyaaos/medgemma-1.5-4b-preventive-qlora")

The prompt format and the exact 7-part output schema are in the repo (src/.../training/formatting.py).

Training

QLoRA on one 8 GB GPU: 4-bit nf4 base, LoRA r=16/α=32 on the language-model projections, 5 epochs, effective batch 8, lr 2e-4 cosine, seq≤2048. Data: 60 synthetic patients, patient-level split 48/6/6 (zero overlap). Final training loss ≈ 0.155.

Evaluation (held-out synthetic test, n=6, automatic metrics)

Finding: fine-tuning reliably fixed the base model's safety-framing gaps and cut hallucinated numbers. After identical fine-tuning, this medical adapter and the non-medical control were indistinguishable (both near-ceiling) — for this task, the payoff rode on the base's general capability more than on medical pretraining. Metrics measure form & faithfulness, not clinical correctness. n=6 + synthetic ⇒ do not overclaim. Full write-up in the repo.

Limitations

Synthetic-data bias; template-derived gold (scores partly reflect format matching); automatic metrics only, no clinical validation; tiny test set; narrow lab/vital scope; can still hallucinate.

License / Notice

This is a Model Derivative of MedGemma (Health AI Developer Foundations). HAI-DEF is provided under and subject to the Health AI Developer Foundations Terms of Use: https://developers.google.com/health-ai-developer-foundations/terms — which govern your use of this adapter together with the base model. You must comply with the HAI-DEF Prohibited Use Policy, must not use it in a way that could cause a health regulatory authority to deem Google a medical-device manufacturer, and must seek health regulatory authorization where applicable. Only the adapter is distributed here; the base weights are not redistributed.