Reza2kn/MiLMMT-46-1B-v1.0-FP8-BLOCK

MiLMMT-46-1B-v1.0 FP8 BLOCK

This repository is a native, reusable FP8 derivative of `xiaomi-research/MiLMMT-46-1B-v1.0`, validated for English-to-Persian translation with vLLM.

Provenance

• —Source revision: 846f74f58800629ccf26481bd7f1ad3903e32e6c
• —Source model.safetensors SHA-256: 60d322f9b330b231a41f1d4fec4d6f3f5ed99c9631b4ea546478d1f335cc27c6
• —Quantized model.safetensors SHA-256: 5708e489c05fc226384d70fea2978e8164395a2ac3a5332197974f1749f09818
• —Quantized tensor bytes: 1,302,155,616

Quantization

• —Exporter: llmcompressor==0.13.0
• —Format: compressed-tensors / float-quantized
• —Scheme: FP8_BLOCK (W8A8 FP8)
• —Weights: static FP8, symmetric 128 x 128 blocks
• —Input activations: dynamic FP8, symmetric groups of 128
• —Left unquantized: lm_head, model.embed_tokens
• —Calibration data: none; model-free PTQ

Gemma 3 ties its embedding and output weights. The embedding table is left unquantized because vLLM's Gemma 3 loader expects an ordinary embedding tensor.

Validation

Validation used all 960 English-to-Iranian-Persian rows from the en-fa_IR configuration of google/wmt24pp. Both BF16 and FP8 used deterministic greedy decoding (temperature=0, top_k=1, maximum 256 new tokens).

Smoke example

Prompt:

Translate this from English to Persian:
English: Siso's depictions of land, water center new gallery exhibition
Persian:

Output:

تصاویر سیسو از زمین و آب، نمایشگاه جدید گالری را تشکیل می‌دهد

Stallion throughput

Measured on one NVIDIA GeForce RTX 5080 Laptop GPU with 16 GB VRAM using vllm==0.20.2, FlashAttention 2, and the selected CutlassFp8BlockScaledMMKernel.

The long-output workload used 307,200 generated tokens and completed in 55.77 seconds. A concurrency-512 attempt exceeded the 16 GB VRAM budget; concurrency 256 is the validated configuration on this machine.

vLLM usage

vllm serve Reza2kn/MiLMMT-46-1B-v1.0-FP8-BLOCK \
  --served-model-name milmmt-46-1b-fp8-block \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --max-num-batched-tokens 32768 \
  --max-num-seqs 256 \
  --enable-chunked-prefill \
  --attention-backend FLASH_ATTN

The quantization format is detected from config.json; do not add an online --quantization conversion flag when serving this saved checkpoint.

Use the source model's translation prompt exactly:

Translate this from <source language name> to <target language name>:
<source language name>: <source language sentence>
<target language name>:

Limitations

• —The quantitative preservation gate above covers English-to-Iranian-Persian. Other supported MiLMMT directions were not promoted by this validation.
• —FP8 output strings are not expected to be byte-identical to BF16. On the locked EN-to-FA gate, quality against human references retained more than 98% on both chrF++ and BLEU.
• —Throughput is workload-, runtime-, and hardware-dependent. The reported numbers are measured receipts, not general hardware claims.
• —The upstream Gemma license and source model limitations continue to apply.

Citation

Please cite the upstream MiLMMT/GemmaX work and identify this repository as a quantized derivative when using it in published results.