Modelpublic

Weidows/laya-multilingual-FP8

sourceHugging Faceapache-2.0updated 18h agoView on Hugging Face

1likes

Model Card

laya-multilingual FP8 (E4M3 per-row)

FP8 quantization of convaiinnovations/laya-multilingual (Apache-2.0). 101 2-D Linear weights in E4M3 with per-row fp32 scales; norms/biases, token embeddings and temperature kept fp16/fp32. 519 MB vs 644 MB original.

Verified (real eval, not weight distance)

XNLI validation, 100 samples x 5 languages, byte-identical questions, fp32 vs fp8:

lang	fp32 acc	fp8 acc	decision agreement	mean conf drift
en	0.930	0.920	0.990	0.009
zh	0.850	0.830	0.960	0.016
ar	0.830	0.830	1.000	0.022
hi	0.650	0.660	0.980	0.024
es	0.820	0.810	0.990	0.015

Accuracy deltas are within +/-0.02 sample noise; decisions agree on 96-100%.

[image]

Use

python

from load_fp8 import load_fp8
agent = load_fp8("Weidows/laya-multilingual-FP8", device="cuda")
res = agent.predict(
    {"body": "Invoice 4411 charged twice, refund please."},
    {"department": {"type": "choice", "instructions": "Which team?",
                    "criteria": {"billing": "invoices, payments, refunds",
                                 "technical": "bugs and outages", "sales": "pricing"}}},
)
print(res["answers"]["department"]["choice"])

Files: model_fp8.safetensors + fp8_scales.json (per-row scales) + load_fp8.py. encoder/, tokenizer/, rl_agent_config.json are unchanged originals.