Modelpublic

rkayaith/Mistral-Medium-3.5-128B-fp8-block

sourceHugging Faceotherupdated 1mo agoView on Hugging Face

0likes28downloads

Model Card

Mistral Medium 3.5 128B — FP8 Block Quantization

This is a dynamic block-FP8 derivative of `mistralai/Mistral-Medium-3.5-128B`.

LLM Compressor dequantized the released static-FP8 linear weights. It then applied the target dynamic block-FP8 recipe.

Quantization details

Property	Value
Format	FP8 E4M3 with block scaling
Weight block size	128×128
Activation group size	128 elements
Weight quantization	Static, per 128×128 block
Activation quantization	Dynamic, per 128-element group
Layers quantized	616 linear weights in 88 decoder layers
Layers skipped	`lm_head`, vision tower, multimodal projector
Checkpoint format	`compressed-tensors`

The quantized layers cover all attention and feed-forward projections in each decoder layer.

Hardware requirements

FP8 block acceleration requires a vLLM backend with block-scaled FP8 linear support.

This checkpoint passed validation on AMD MI350X (CDNA4, gfx950) with ROCm 7.2.3.

Note: Other hardware configurations have not been validated with this checkpoint.

Usage with vLLM

python

from vllm import LLM, SamplingParams

llm = LLM(
    model="rkayaith/Mistral-Medium-3.5-128B-fp8-block",
    language_model_only=True,
)

outputs = llm.generate(
    ["Tell me about AMD MI350X"],
    SamplingParams(max_tokens=200, temperature=0.7),
)
print(outputs[0].outputs[0].text)

Use this command to start a language-only server:

bash

vllm serve rkayaith/Mistral-Medium-3.5-128B-fp8-block \
  --language-model-only

Comparison variants

Variant	Repository	Format
Static FP8	`mistralai/Mistral-Medium-3.5-128B`	Released static FP8
Dynamic block FP8	`rkayaith/Mistral-Medium-3.5-128B-fp8-block`	FP8 E4M3, 128×128 weight blocks, dynamic 128-element activation groups
Block MXFP8	`rkayaith/Mistral-Medium-3.5-128B-mxfp8`	OCP MXFP8, 32-element groups

License

This derivative uses the source model's `Modified MIT License`.

Modifications: The released static-FP8 linear weights were dequantized and requantized to dynamic block FP8. No fine-tuning occurred.