Modelpublic

RapidMark/Qwen2.5-1.5B-E4M3-GGUF

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face

0likes11downloads

Model Card

Qwen2.5-1.5B-Instruct E4M3 (fp8) GGUF

Qwen2.5-1.5B-Instruct quantized to E4M3 (OCP e4m3fn fp8) for llama.cpp. Test model for the CPU e4m3 quant type.

Block layout matches Q80: one fp16 scale + 32 one-byte elements per 32-value block. Q80 spends the byte on a linear int8; e4m3 spends it on a float (1 sign / 4 exponent / 3 mantissa).

How it was made

Built from the f16 GGUF with llama.cpp (branch cloudhands/fp8-cpu, HEAD 8a02ad204):

llama-quantize qwen2.5-1.5b-f16.gguf qwen2.5-1.5b-e4m3.gguf E4M3

Evaluation

Qwen2.5-1.5B-Instruct, wikitext-2-raw test (full, 584 chunks, n_ctx 512), pure CPU.

type	size	PPL	vs f16	Mean KLD	Median KLD	Max KLD	same-top-token
f16	2.88 GiB	10.134	-	-	-	-	-
q8_0	1.53 GiB	10.168	+0.33%	0.00200	0.00158	0.213	97.33%
e4m3	1.48 GiB	10.248	+1.12%	0.01146	0.00937	0.846	93.64%

llama-bench, pure CPU, 64 threads:

type	pp512 (t/s)	tg128 (t/s)
f16	528.3	19.2
q8_0	499.3	33.1
e4m3	253.2	34.6

Notes

Reference weights for testing, not tuned for downstream use.