bisonnetworking/medgemma-health-chat-gguf

MedGemma Health Chat — GGUF Quantized

GGUF quantized versions of the MedGemma Health Chat model, fine-tuned from google/medgemma-1.5-4b-it for multi-persona health chat conversations.

Available Quantizations

Quantization Method

These GGUF files were produced using llama.cpp's official conversion and quantization pipeline — the same method used by bartowski and the llama.cpp community.

Process

1. 1.Base model: The merged 16-bit model (bisonnetworking/medgemma-health-chat-merged) was downloaded from HF Hub
2. 2.Conversion: python convert_hf_to_gguf.py converted the HuggingFace safetensors to GGUF F16 format (444 tensors, 7.24 GB)
3. 3.Quantization: llama-quantize applied K-quant methods to produce Q4KM and Q8_0 from the F16 baseline

Why K-quants?

K-quant methods (Q4KM, Q8_0) use mixed-precision tensor quantization — different tensor types get different bit depths based on their sensitivity:

• —Attention weights (qproj, kproj, oproj): quantized to Q4K
• —Value and FFN down weights (vproj, ffndown): quantized to Q6_K (higher precision)
• —FFN gate/up weights: quantized to Q4_K
• —Norm weights: kept at F32 (full precision)

This mixed approach preserves more quality than uniform quantization at the same average bit rate. Q4KM specifically balances size and quality — it's the community standard for 4-bit GGUF.

No imatrix

These quantizations were produced without an importance matrix (imatrix). For a 4B model at Q4KM, the quality difference is minimal. If you need maximum quality at low bitrates (Q2K/Q3K), consider generating an imatrix from your calibration data and re-quantizing.

How to Use

With llama.cpp

# Download the Q4_K_M (recommended)
huggingface-cli download bisonnetworking/medgemma-health-chat-gguf \
  medgemma-health-chat-Q4_K_M.gguf \
  --local-dir ./models

# Interactive chat
./llama-cli -m ./models/medgemma-health-chat-Q4_K_M.gguf \
  --interactive \
  -ngl 99 \
  -c 2048

# Single prompt with system message
./llama-cli -m ./models/medgemma-health-chat-Q4_K_M.gguf \
  -sys "You are a board-certified Primary Care Physician. Provide direct, clinical guidance. No tables. No AI disclaimers." \
  -p "I've had a sore throat and low-grade fever for 3 days. What should I do?" \
  -ngl 99 \
  -c 2048

With Ollama

# Create a Modelfile
cat > Modelfile << 'EOF'
FROM ./medgemma-health-chat-Q4_K_M.gguf
TEMPLATE """{{ if .System }}<start_of_turn>user
{{ .System }}<end_of_turn>
{{ end }}<start_of_turn>user
{{ .Prompt }}<end_of_turn>
<start_of_turn>model
"""
PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER num_ctx 2048
PARAMETER stop "<end_of_turn>"
EOF

# Create and run
ollama create medgemma-health-chat -f Modelfile
ollama run medgemma-health-chat "I've had a sore throat for 3 days. What should I do?"

With LM Studio

1. 1.Download the GGUF file to ~/.cache/lm-studio/models/bisonnetworking/medgemma-health-chat-gguf/
2. 2.Open LM Studio and select the model
3. 3.Set GPU offload layers to maximum (all layers fit on most modern GPUs)
4. 4.Set context length to 2048
5. 5.Start chatting

With llama-cpp-python

from llama_cpp import Llama

llm = Llama(
    model_path="./medgemma-health-chat-Q4_K_M.gguf",
    n_ctx=2048,
    n_gpu_layers=99,  # Apple Metal or CUDA offload
    chat_format="gemma3",
)

messages = [
    {
        "role": "system",
        "content": "You are a board-certified Primary Care Physician. Provide direct, clinical guidance. No tables. No AI disclaimers.",
    },
    {
        "role": "user",
        "content": "I've had a sore throat and low-grade fever (100.4) for 3 days. No cough, no swollen lymph nodes. What should I do?",
    },
]

response = llm.create_chat_completion(
    messages=messages,
    max_tokens=256,
    temperature=0.7,
    top_p=0.9,
)
print(response["choices"][0]["message"]["content"])

As an OpenAI-Compatible Server

# Start server
./llama-server -m ./medgemma-health-chat-Q4_K_M.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  -ngl 99 \
  -c 2048

# Use with any OpenAI client
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "medgemma-health-chat",
    "messages": [
      {"role": "system", "content": "You are a Primary Care Physician..."},
      {"role": "user", "content": "I have a sore throat. What should I do?"}
    ],
    "max_tokens": 256,
    "temperature": 0.7
  }'

Model Details

• —Base model: google/medgemma-1.5-4b-it (Gemma 3 4B architecture)
• —Architecture: Gemma3ForConditionalGeneration
• —Parameters: 4,365,656,432 (4.4B)
• —Layers: 34
• —Embedding dimension: 2560
• —Attention heads: 8 (4 KV heads, GQA)
• —Context length: 131,072 (use 2048 for health chat)
• —Chat template: Gemma3 (startofturn/endofturn format)

Training

See the merged model card for full training details. Summary:

• —LoRA fine-tune (r=32, alpha=64) on 49,500 health chat conversations
• —2 epochs, 12,376 steps, H100 80GB via Modal
• —Final training loss: 0.7981, eval loss: 0.8779
• —6 medical personas: primary care, internal medicine, clinical nutritionist, exercise specialist, best doctor, chronic health

Limitations

• —GGUF quantization introduces minor precision loss vs the F16 original
• —Q4KM may occasionally produce slightly different phrasing than the full-precision model
• —The Gemma3 chat template must be used correctly — mismatched templates will produce poor output
• —Not a substitute for professional medical advice
• —MedGemma base model is gated — users need approved access to the original model

Related Models

• —LoRA adapter — 250MB, for use with PEFT
• —Merged 16-bit — 8.6GB, full precision
• —Training dataset — 49,500 conversations

Model Card Contact

bisonnetworking