Demondiablo/medgemma-4b-it-int4-rtn-w4a16

MedGemma 4B-IT (INT4 RTN W4A16)

This is an INT4 RTN W4A16 (Round-To-Nearest 4-bit Weight-Only) quantized version of google/medgemma-4b-it created using llm-compressor and formatted in compressed-tensors.

Quantization Details

• —Base Model: google/medgemma-4b-it
• —Algorithm: Round-To-Nearest (RTN) data-free quantization
• —Quantization Scheme: W4A16
• —Weights: 4-bit integer (int4), symmetric, per-group scaling (group_size=128)
• —Activations: 16-bit float / bfloat16 (unquantized)
• —Preserved Precision (BF16): lm_head, embed_tokens, multi_modal_projector, and vision tower components are preserved in native BF16 for high clinical fidelity.
• —Hardware Platform: Quantized and evaluated on NVIDIA RTX PRO 6000 Blackwell Server Edition (98 GB VRAM).
• —Compressed Checkpoint Size: ~3.6 GB (down from ~8.6 GB bfloat16).

High-Performance Deployment with vLLM

vLLM natively accelerates compressed-tensors W4A16 checkpoints:

from vllm import LLM, SamplingParams

model_name = "Demondiablo/medgemma-4b-it-int4-rtn-w4a16"

# Initialize vLLM engine
llm = LLM(
    model=model_name,
    trust_remote_code=True,
    max_model_len=4096,
)

prompt = "Extract medications: Capsule Desula 21 HS Tab 81 2 mg 1 HS Tab Telista CL 1 BD, Thyronorm 50 mg OD."
messages = [{"role": "user", "content": prompt}]

sampling_params = SamplingParams(
    temperature=0.2,
    max_tokens=512,
    top_p=0.95,
)

outputs = llm.chat(messages=messages, sampling_params=sampling_params)
print(outputs[0].outputs[0].text)

Evaluation Artifacts & JSON Output

• —model.safetensors: INT4 compressed weights
• —config.json: Architecture parameters and quantization_config
• —recipe.yaml: Reproducible LLM Compressor recipe
• —medgemma-4b-it-int4-rtn-w4a16.json: Full benchmark evaluation results with generation latency and metrics
• —extractions_medgemma-4b-it-int4-rtn-w4a16.json: Parsed clinical medication extractions across benchmark audio transcripts