Modelpublic

darthcrawl/granite-guardian-4.1-8b-mlx-4bit

sourceHugging Faceapache-2.0updated 5mo agoView on Hugging Face

0likes42downloads

Model Card

granite-guardian-4.1-8b-mlx-4bit

MLX 4-bit quant of ibm-granite/granite-guardian-4.1-8b. IBM's safety classifier; flags harmful content across categories (harm, social bias, jailbreak, violence, profanity, sexual content, unethical behavior).

Disk: ~4.4 GB. Active memory while serving: ~5 GB plus KV cache.

Quantization

bash

mlx_lm.convert --hf-path ibm-granite/granite-guardian-4.1-8b \
  --mlx-path ./granite-guardian-4.1-8b-mlx-4bit -q --q-bits 4

Usage

Granite Guardian uses a structured prompt to classify a piece of input/output text. The model returns "Yes" or "No" plus a confidence score. See the base model card for the prompt template (it's risk-category-specific).

For a quick check via mlx_lm.server's chat completions endpoint, send the structured prompt as the user message and read the assistant token.

Recommended sampling

Classification, not creative output:

yaml

temperature: 0.0
max_tokens: 16

Greedy decoding is appropriate. Bail early since the response is short.

License

Apache-2.0 from base.