Datasetpublic

Zeraix/imparo-benchmarks

Imparo inference benchmarks Published performance measurements for Imparo, a hardware- and workload-adaptive LLM inference engine. This dataset makes the project's published benchmark table available in a machine-readable form. It contains 48 aggregate results across four engines and 12 model/workload combinations, not 48 independent benchmark runs or a training corpus. Project · Pinned source table · Zeraix organization Apple M3 Pro — published 2026-09-07… See the full description on the dataset page: https://huggingface.co/datasets/Zeraix/imparo-benchmarks.

sourceHugging Faceapache-2.0updated 19d agoView on Hugging Face

0likes43downloads

Dataset Card

Imparo inference benchmarks

Published performance measurements for Imparo, a hardware- and workload-adaptive LLM inference engine.

This dataset makes the project's published benchmark table available in a machine-readable form. It contains 48 aggregate results across four engines and 12 model/workload combinations, not 48 independent benchmark runs or a training corpus.

Project · Pinned source table · Zeraix organization

Apple M3 Pro — published 2026-09-07

—Models: Gemma 4 E4B (UD-Q4KXL) and LFM2.5-2.6B (Q8_0).
—Engines: Imparo, llama.cpp, oMLX, and rapid-mlx.
—Configuration reported by the source: f16 KV cache, 512-token prefill chunks, thinking disabled on every engine.
—Method: two interleaved rounds in the same session; the client sends the same prompt to each engine in turn. The table reports the median in tokens/second.
—Timing: prefill throughput is derived from time to first token; decode throughput is derived from the token stream. These are client-observed measurements, not engine-reported kernel throughput.

Higher tokens/second is better. Compare engines within the same row, not across different models or prompt lengths.

Model	Prompt tokens	Phase	Imparo	llama.cpp	oMLX	rapid-mlx
Gemma 4 E4B	449	prefill	932	556	590	830
Gemma 4 E4B	5651	prefill	1063	570	904	960
Gemma 4 E4B	16191	prefill	1006	522	888	914
LFM2.5-2.6B	455	prefill	1023	933	675	857
LFM2.5-2.6B	5963	prefill	1060	980	964	1007
LFM2.5-2.6B	17123	prefill	972	900	921	957
Gemma 4 E4B	449	decode	46.6	40.5	43.9	42.8
Gemma 4 E4B	5651	decode	44.2	38.5	41.8	40.5
Gemma 4 E4B	16191	decode	40	34.5	37.9	36.7
LFM2.5-2.6B	455	decode	46.6	43.9	46.8	44.6
LFM2.5-2.6B	5963	decode	45.1	42.4	44.6	42.6
LFM2.5-2.6B	17123	decode	42.7	40	40.8	39.6

Scope and limitations

These values are transcribed from the public source table, not a new benchmark run or an independent reproduction. They describe the reported hardware, models, and configurations only.

The source table does not provide full per-round traces, exact prompt text, all tested engine commit IDs, RAM capacity, or OS version. The linked README commit pins the documentation snapshot, not the versions of every engine tested. Two rounds do not establish statistical significance; small differences should not be treated as definitive rankings. In particular, LFM2.5-2.6B short-prompt decode is 46.6 tok/s for Imparo and 46.8 tok/s for oMLX, described as a tie in the source.

M4 Pro: the project's 2026-09-10 update reports validation reproducing the optimization benefits observed on M3 Pro. No complete four-engine M4 Pro numerical table is included here. M3 Pro measurements must not be relabeled as M4 Pro results.

Data format

The JSONL file has one record per model, prompt length, phase, and engine:

Field	Meaning
measurement_id	Stable identifier for this published aggregate record.
hardware	Hardware label reported in the source.
model	Model name reported in the source.
quantization	Quantization label reported for the model.
prompt_tokens	Prompt length reported in the source table.
phase	prefill or decode.
engine	Engine being measured.
tokenspersecond	Published median throughput in tok/s.

The configuration name uses the publication date, not a claimed exact execution timestamp. Future hardware results should be added as separate configurations, preserving earlier measurements and their provenance.

Contribute a reproducible result

Open a discussion or an Imparo issue. Include hardware and RAM, OS, model revision and quantization, engine commits, launch commands, prompts and output lengths, cache/warm-up settings, repeated runs, and correctness checks.

Attribution and license

Maintained by Zeraix. The source table comes from the Apache-2.0-licensed Imparo repository. This repository's documentation and benchmark-table transcription use Apache-2.0. No model weights, private logs, or user conversations are included. Upstream models and engines retain their own licenses.