gratex/Mistral-Small-3.2-24B-Instruct-2506-sbvr-compliance-W4A16

Mistral-Small-3.2-24B-Instruct-2506-sbvr-compliance-W4A16

Model description

This is a W4A16 quantized version of `gratex/Mistral-Small-3.2-24B-Instruct-2506-sbvr-compliance-unresolvable-pattern-finetune-v1`, which is a fine-tuned Mistral-Small-3.2-24B-Instruct-2506 designed to improve the evaluation of SBVR (Semantics of Business Vocabulary and Business Rules) rules.

The quantization was performed using AutoRound 0.15.0 with SignRoundV2 optimization, reducing the model size from 45 GB (BF16) to 15 GB while maintaining high fidelity to the original model. The model supports both text-only and image+text (vision) inference.

The multimodal projector tensors are stored under their Mistral-native names (multi_modal_projector.w_in, multi_modal_projector.w_out, pre_mm_projector_norm, patch_merger.merging_layer) so that vLLM's Mistral-native loading path matches them correctly; these tensors are BF16 and are not quantized.

Quantization configuration

Calibration dataset

The calibration dataset was constructed from multiple domain-specific sources to ensure the quantized model retains accuracy across all relevant domains:

All 512 calibration samples were concatenated to a minimum of 4,096 tokens each (20,480 characters) to maximize the calibration signal per sample. The dataset is published alongside this model as calibration_dataset.jsonl.

Quantization script

The quantization script is published as quantize.py in this repository.

Evaluation results

WikiText-2 Perplexity

Standard WikiText-2-raw-v1 benchmark with sliding window (stride=512, max_length=2048, 590 windows, 301,616 tokens):

Comprehensive evaluation

30 domain-specific prompts (SBVR, Slovak, English, code, reasoning) comparing token-level agreement, KL divergence, and generation fidelity against the BF16 baseline:

Per-category PPL

Usage

vLLM (recommended for both text and vision)

This model requires vLLM's Mistral-native loading mode. Deploy with config_format: mistral and tokenizer_mode: mistral (the Tekken tokenizer is shipped as tekken.json; using the default HF tokenizer mode produces garbled output):

docker run -d --name vllm-mistral-sbvr \
  --runtime=nvidia --gpus '"device=0"' \
  -p 8000:8000 \
  -v /path/to/model:/workspace/model \
  --ipc=host --shm-size=16g \
  vllm/vllm-openai:cu129-nightly \
  --model /workspace/model \
  --served-model-name mistral-24b-sbvr-w4a16 \
  --config-format mistral --tokenizer-mode mistral \
  --quantization auto_round \
  --dtype bfloat16 \
  --trust-remote-code

Both text-only and image+text (vision) requests are supported:

import base64, requests

img_b64 = base64.b64encode(open("image.jpg", "rb").read()).decode()

# Text
requests.post("http://localhost:8000/v1/chat/completions", json={
    "model": "mistral-24b-sbvr-w4a16",
    "messages": [{"role": "user", "content": "Evaluate this SBVR rule: ..."}],
    "max_tokens": 512,
})

# Vision
requests.post("http://localhost:8000/v1/chat/completions", json={
    "model": "mistral-24b-sbvr-w4a16",
    "messages": [{"role": "user", "content": [
        {"type": "text", "text": "Describe this image."},
        {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{img_b64}"}},
    ]}],
    "max_tokens": 512,
})

transformers

The projector tensors use Mistral-native names for vLLM compatibility. transformers 5.x loads Mistral3 checkpoints without a name-conversion layer, so the projector tensors need manual renaming (w_in→linear_1, w_out→linear_2, pre_mm_projector_norm→norm, patch_merger→multi_modal_projector.patch_merger.merging_layer) when loading this model with transformers. For production use, the vLLM path above is the recommended and tested route.

Model details

HELM comparison: fine-tuned BF16 vs W4A16

The BF16 fine-tuned source model and this W4A16 quantized version were evaluated side by side on Stanford HELM benchmarks (crfm-helm >= 0.5.0, 1,000 instances per scenario, served via an OpenAI-compatible endpoint).

General benchmarks

MMLU Pro

Summary averages

BBQ bias

Original model

This model is a quantized version of `gratex/Mistral-Small-3.2-24B-Instruct-2506-sbvr-compliance-unresolvable-pattern-finetune-v1`. Please refer to the original model card for detailed information about the fine-tuning methodology, training configuration, and HELM evaluation results.