Modelpublic

mbehr90/multilingual-e5-large-instruct-fp8

sourceHugging Facemitupdated 25d agoView on Hugging Face

0likes296downloads

Model Card

multilingual-e5-large-instruct-fp8

FP8_DYNAMIC quantization of intfloat/multilingual-e5-large-instruct, produced with llm-compressor 0.13.0 and measured end to end on NVIDIA H100 80 GB with vLLM 0.26.0.

Only the Linear layers of the encoder are quantized; the vocabulary, position and token-type embeddings, the pooler and any classification head stay in bf16.

Measured

variant	size	SciFact nDCG@10	STS-B de ρ	STS17 en-de ρ	texts/s (bs=256)
bf16 source	1068 MiB	0.7204	0.8347	0.8525	1488
FP8	802 MiB	0.7164	0.8349	0.8526	1511
NVFP4	675 MiB	0.7036	0.8371	0.8440	–

Retrieval and STS are unchanged against bf16. Size is −25%. Throughput is unchanged — the gain is memory only. If you want the smaller checkpoint, the NVFP4 sibling is 675 MiB for 1.7 pp nDCG@10.

SciFact: 300 queries over 5183 documents, nDCG@10. STS-B de: mteb/stsbmultimt (1379 pairs). STS17 en-de: mteb/sts17-crosslingual-sts (250 pairs). Reranking: 1 positive + 19 sampled negatives per query. Throughput over 4096 multilingual texts through the vLLM OpenAI endpoint.

Serving

bash

vllm serve mbehr90/multilingual-e5-large-instruct-fp8 --runner pooling