rockyaaos/gemma-3-4b-preventive-qlora

Gemma 3 4B · Preventive-Health QLoRA adapter (v1, control)

Research / education only. Synthetic training data only. NOT a diagnostic tool, not clinical decision support, not a medical device. Do not use with real patient data or PHI.

A LoRA/QLoRA adapter (≈30M params, ~57 MB) trained on top of google/gemma-3-4b-it. It is the experimental control for a study of whether medical continued-pretraining helps: same base family as MedGemma but without medical pretraining, fine-tuned with an identical recipe on identical synthetic data. Not a standalone model — you need separate access to the gated base.

Study: github.com/rockyzl/preventive-health-model-lab. Medical arm: `medgemma-1.5-4b-preventive-qlora`.

Usage

import torch
from transformers import AutoTokenizer, BitsAndBytesConfig, Gemma3ForConditionalGeneration
from peft import PeftModel

base = "google/gemma-3-4b-it"               # accept its license on HF first
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16)
tok = AutoTokenizer.from_pretrained(base)
model = Gemma3ForConditionalGeneration.from_pretrained(base, quantization_config=bnb, device_map="cuda")
model = PeftModel.from_pretrained(model, "rockyaaos/gemma-3-4b-preventive-qlora")

Training

QLoRA on one 8 GB GPU: 4-bit nf4 base, LoRA r=16/α=32 on the language-model projections, 5 epochs, effective batch 8, lr 2e-4 cosine, seq≤2048. Data: 60 synthetic patients, patient-level split 48/6/6 (zero overlap). Final training loss ≈ 0.157.

Evaluation (held-out synthetic test, n=6, automatic metrics)

Finding: after identical fine-tuning, this non-medical control was indistinguishable from the medical MedGemma adapter (both near-ceiling; the small gap is within n=6 noise). The useful read: a capable base plus task fine-tuning carried the result — medical pretraining was not the deciding factor for this task. Metrics measure form & faithfulness, not clinical correctness. n=6 + synthetic ⇒ do not overclaim.

Limitations

Synthetic-data bias; template-derived gold; automatic metrics only; no clinical validation; tiny test set; narrow lab/vital scope; possible hallucination.

License / Notice

This is a Model Derivative of Gemma. Gemma is provided under and subject to the Gemma Terms of Use found at https://ai.google.dev/gemma/terms, and use is subject to the Gemma Prohibited Use Policy. Only the adapter is distributed here; the base weights are not redistributed.