Datasetpublic

blairducrayoppat/npu-specdecode-lunarlake

Speculative decoding draft-device characterization on Intel Lunar Lake (OpenVINO GenAI) Reference performance data for heterogeneous speculative decoding with OpenVINO GenAI: a GPU target model (Qwen3-14B INT4) paired with a small draft model (Qwen3-0.6B INT4) run on the CPU vs the NPU vs the GPU, plus standalone draft-only throughput per device. Measured on a single Intel Core Ultra 7 258V (Lunar Lake) laptop. Headline: putting the draft on the NPU is a net slowdown (0.55–0.74×… See the full description on the dataset page: https://huggingface.co/datasets/blairducrayoppat/npu-specdecode-lunarlake.

sourceHugging Facecc-by-4.0updated 3mo agoView on Hugging Face

0likes21downloads

Dataset Card

Speculative decoding draft-device characterization on Intel Lunar Lake (OpenVINO GenAI)

Reference performance data for heterogeneous speculative decoding with OpenVINO GenAI: a GPU target model (Qwen3-14B INT4) paired with a small draft model (Qwen3-0.6B INT4) run on the CPU vs the NPU vs the GPU, plus standalone draft-only throughput per device. Measured on a single Intel Core Ultra 7 258V (Lunar Lake) laptop.

Headline: putting the draft on the NPU is a net slowdown (0.55–0.74× of GPU-only throughput); the same draft on the CPU is a 1.35× speedup. Root cause: the NPU runs the small draft model ~3.6× slower per token than the CPU, and speculative decoding only pays off when the draft is cheap. This corroborates OpenVINO's documented guidance (CPU draft for a GPU/CPU target, or both models on the NPU) — it is reference characterization, not a bug report.

—Narrative write-up: https://github.com/openvinotoolkit/openvino/discussions/36484
—Related (closed) issue: https://github.com/openvinotoolkit/openvino/issues/34450

Hardware

Component	Detail
CPU	Intel Core Ultra 7 258V (Lunar Lake)
GPU	Intel Arc 140V (Xe2, 16 GB)
NPU	Intel AI Boost
OS	Windows 11 Pro build 26200 (25H2)
NPU driver	32.0.100.4778
GPU driver	32.0.101.8826
OpenVINO	2026.1.0
OpenVINO GenAI	2026.1.0.0

Models

—Target: Qwen3-14B, INT4 (asymmetric per-group — fine on GPU).
—Draft: Qwen3-0.6B, INT4 symmetric (--sym --group-size 128 --ratio 1.0). Symmetric is required for the NPU (per-group asymmetric INT4 is unsupported there). Same draft used on every device.
—Target and draft share the Qwen3 tokenizer (required for speculative decoding).

Methodology

Greedy, ignore_eos=True, fixed 128-token budget, 1 warm-up + 3 measured reps, each config in a fresh process. num_assistant_tokens (NAT) is the stateful pipeline's adaptive seed, not a fixed per-round proposal count. Metrics from PerfMetrics / SDPerModelsPerfMetrics; acceptance_rate = accepted / draft-proposed. Construct/compile time is excluded from throughput. Single prompt — acceptance numbers are illustrative; the throughput/per-token-latency result is the robust, device-physics finding.

Results

Standalone draft-only (Qwen3-0.6B-sym, no speculative decoding)

device	throughput	ms/token
CPU	106.8 tok/s	9.4
GPU	150.0 tok/s	6.7
NPU	29.2 tok/s	34.3

Speculative decoding (GPU target + draft)

draft device	NAT seed	throughput	speedup	acceptance
none (GPU only)	—	12.2 tok/s	1.00×	—
CPU	3	16.5 tok/s	1.35×	0.40
CPU	5	14.7 tok/s	1.20×	0.32
NPU	1	9.0 tok/s	0.74×	0.60
NPU	3	7.1 tok/s	0.58×	0.35
NPU	5	6.7 tok/s	0.55×	0.32

Files

—npu_specdecode_perf.jsonl — one record per config (standalone + spec-decode); programmatic import.
—npu_specdecode_perf.csv — flattened (env_* / meas_* prefixes).

Not measured (explicit)

Single machine, single prompt, n=3; greedy only; default NPU prompt/response budgets; no co-resident memory cost; no subjective quality beyond a greedy token-identity check. Behavior may differ on OpenVINO 2026.2/master.

License

CC-BY-4.0 (adjust to your preference before publishing). Author: Blair DuCray-Oppat (GitHub @blairducrayoppat).

Citation

NPU vs CPU draft speculative-decoding characterization on Intel Lunar Lake (OpenVINO GenAI 2026.1), 2026. https://github.com/openvinotoolkit/openvino/discussions/36484