blairducrayoppat/openvino-arc140v-lunarlake

OpenVINO local-inference on an Intel Arc 140V (Lunar Lake) iGPU

Reference performance data for running local models on a single Intel Core Ultra 7 258V (Lunar Lake) laptop with the integrated Intel Arc 140V (Xe2) GPU, via OpenVINO. All inference runs on the iGPU; the NPU stays idle throughout, confirmed by the telemetry here.

This is reference characterization shared by a non-expert contributor — careful measurements on one machine, offered so others can compare and correct, not a definitive benchmark. Every number is scoped to the hardware, driver, runtime, and model stated here. Feedback and corrections are very welcome — if something looks off or I've measured the wrong thing, please open a discussion.

Start here: OFFLOAD_RATIO unlocks a 35B MoE on a 32 GB laptop

OpenVINO 2026.3.0, released weeks ago on 2026-08-04, added GPU MoE expert offloading. The best thing in this dataset is what that brand-new feature does to a model this laptop could not practically run before it:

OpenVINO/Qwen3.6-35B-A3B-int4-ov — a 19.7 GB vision-language MoE — goes from 1.44 tok/s fully resident to 21.4 tok/s with OFFLOAD_RATIO=20. 14.9×, on the same machine, same driver, same model, from one property.

The mechanism is straightforward once measured. Fully resident, the model plus its KV cache plus the runtime plus the OS push committed memory to 29.90 GiB against 31.3 GiB physical, and the machine pages on every token — 693 ms per token, steadily (N=3, sd 1.0%). Offloading ~20% of the routed expert weights drops committed to 25.65 GiB, the paging stops, and decode runs at full speed.

OFFLOAD_RATIO defaults to 0 (everything resident), so this is opt-in: on a unified-memory machine of this size it is worth setting deliberately rather than leaving at the default. There is currently no published sizing guidance for how to pick it on integrated GPUs, which is the substance of the upstream request in openvino#37737 — this dataset's curves are offered partly as input to that.

Two things to know before relying on it:

• —It trades first-token latency for decode speed. TTFT goes the other way: 2.1 s resident versus 21.9 s at ratio 20, because the first token has to stream experts in. Setting the two curves equal, the break-even is around 31 generated tokens — below that, staying resident is genuinely faster end-to-end; above it, offloading pulls away, and by 256 tokens it is roughly 5× (about 34 s versus about 177 s measured).
• —The floor is uncharacterised. Ratios 0 and 20–50 were measured; nothing in between. Ratio 20 is the lowest setting known to work here, not the lowest that does.

Full curve, method, and caveats in Study 1 below.

The two studies in this dataset

This page holds two independent studies, run on the same laptop but on different OpenVINO versions. They do not share a stack, and each has its own methodology, hardware notes, and caveats — read the ones belonging to the study you are citing.

If you are citing a number from this page, check which study it belongs to. The two were measured on different OpenVINO releases and are not comparable across them.

Study 1 — GPU MoE expert offloading on 32 GB unified memory (2026-08-28 / 2026-08-29)

Stack: openvino 2026.3.1 + openvino-genai 2026.3.1.0 (pip wheels), GPU driver 32.0.101.8826 (both failure modes re-validated on 32.0.101.8991), Windows 11 Pro 26200. Data under `moe_offload_2026-08-28/`.

Overview and stack

OpenVINO 2026.3 added GPU MoE expert offloading: OFFLOAD_RATIO=N keeps N% of eligible expert weights off-GPU, streamed on demand through an LRU cache. This study measures it on the same machine as everything above (Core Ultra 7 258V / Arc 140V, 32 GB unified, driver 32.0.101.8826 — both failure modes re-validated on 32.0.101.8991 — Windows 11 Pro 26200) with openvino 2026.3.1 + openvino-genai 2026.3.1.0 (pip) and the INT8 (30.8 GB) and INT4 (16.3 GB) Qwen3-30B-A3B models — a model deliberately bigger than what fits resident in INT8, so offloading is load-bearing. Same caveats as the rest of this dataset: one machine, careful measurement by a non-expert contributor (all timing on one driver; the two failure modes re-validated on a second); corrections welcome.

Attention-backend note (added 2026-08-29). The LLM and VLM numbers in this study come from GenAI's default PagedAttention (continuous-batching) backend — verified from compile logs by re-running the same configurations at OPENVINO_LOG_LEVEL=3, where the runtime prints "Model: LLM with Paged Attention" (the raised-log pylogs published in eagle3/ carry the same line for the 30B). An explicit ATTENTION_BACKEND=SDPA A/B on the VLM (text arm, ratio 30, back-to-back) decoded 11.9% slower than the PagedAttention default (TPOT 64.15 ms vs 57.34 ms; 15.59 vs 17.44 tok/s) — the default was also the faster backend here.

What offloading makes possible on this machine

Three results, all on hardware that costs what a laptop costs:

• —A 35B vision-language MoE becomes usable. OpenVINO/Qwen3.6-35B-A3B-int4-ov (19.7 GB) goes from 1.44 tok/s fully resident to 21.4 tok/s at OFFLOAD_RATIO=20 — 14.9× — because offloading ~20% of the routed experts drops committed memory from 29.90 GiB to 25.65 GiB and the machine stops paging. Details and the full ratio curve are in the 35B vision-language MoE section below.
• —*Offloading can be faster than staying resident. On the INT4 Qwen3-30B-A3B, ratio 30 measured 29.66 tok/s (N=3) against 25.94 tok/s fully resident (N=5) — +14%*. Giving the GPU less to hold was not a tax here; it was a win. (A cross-chain comparison rather than a single back-to-back A/B, but the direction is consistent across every run.)
• —It roughly doubles the context you can hold. Offloading 30% frees ~4.7 GiB of GPU-shared residency, and an 8,136-token prompt at ratio 30 fits in 24.0 GiB committed with a clean two-generate session. Estimated practical ceiling moves from ~7K–17K to ~26K–40K tokens (extrapolated from the 8K measurement; arithmetic in the report).

The same study also turned up two reproducible bugs, both reported upstream with full evidence — they are in the findings below, and they are the reason every INT8 number here uses a fresh process per generate.

Key findings in full — including two bugs reported upstream

• —Two reproducible failure modes, both reported upstream (issues: #37736 hang, #37737 overcommit; full evidence in moe_offload_2026-08-28/):
• —INT8-specific second-generate hang — the first generate() on an LLMPipeline completes, the second on the same instance never returns (generation-length-independent, consistent with a prefill/dispatch-phase hang, ~1 CPU core spinning, memory flat, 5/5 reproductions across ratios 75 and 60 and drivers 32.0.101.8826/.8991 + once under telemetry). Reproduces with the 1054-token prompt, NOT with the 54-token prompt (threshold uncharacterized). Re-creating the pipeline clears it; INT4 is unaffected in every configuration tested, including at the matched ratio 75. Every INT8 number below therefore comes from a fresh process per generate (methodology in the report).
• —Unified-memory overcommit at both ratio extremes — ratio 100 overshoots the driver's 25.2 GiB pool during prefill (shared-usage peak 27.1 GiB) and pushes committed to a 40.6 GiB peak (pagefile thrash, no output; re-validated on driver 32.0.101.8991 in a 240 s-capped run — same no-output overcommit, committed 39.3 GiB); ratio 0 can't fit 28.6 GiB of INT8 weights in the same pool. No cache-bounding property exists in 2026.3.1 (full SUPPORTED_PROPERTIES dump included). Usable INT8 envelope on 32 GB: ratio ~25-75.
• —INT8 decode vs ratio (1054-tok prompt, 256 new tokens, greedy, medians): 3.26 tok/s @75 (N=5, sd 0.8%) → 4.68 @60 (N=3, clean re-run) → 7.11 @40 (N=3) — but TTFT rises as offload falls (short-prompt arm: 18.7 s @75 → 22.4 → 28.7 → 32.1 s @25): consistent with a fixed per-instance cost of uploading the resident expert set dominating first-token time; with the hang-forced fresh-process workaround it is paid on every generate. Steady-state INT8 TTFT with long prompts is unobservable until the hang is fixed (the short-prompt case IS observable: 8.8 s warm vs 18.7 s fresh) — the INT4 control shows what the long-prompt case should look like (warm second-generate TTFT 250-994 ms).
• —INT4 is the practical configuration on this machine: 25.94 tok/s resident (N=5, TTFT 1.19 s) and — unexpectedly — 29.66 tok/s at `OFFLOAD_RATIO=30` (N=3, +14%, cross-chain comparison, direction consistent across all runs). An 8,136-token prompt at ratio 30 fits in 24.0 GiB committed with a clean two-generate session (prefill TTFT 20.2 s, then 250 ms warm). Offloading 30% frees ~4.7 GiB of GPU-shared residency — roughly double the context headroom of fully-resident (est. 26K-40K vs 7K-17K max practical tokens; extrapolated from the 8K measurement, arithmetic in the report).
• —Streaming offload is disk-contention-sensitive (labeled negative result): a concurrent 16 GB download to the same NVMe cut ratio-60 decode from 4.68 to 3.79 tok/s median and swung TTFT 32→47 s.
• —EAGLE-3 speculative decoding cannot run against this MoE target at all (2026-08-29 addendum; reported upstream: [openvino.genai#4390](https://github.com/openvinotoolkit/openvino.genai/issues/4390); full evidence in `moe_offload_2026-08-28/eagle3/`). On openvino-genai 2026.3.1.0, attaching an EAGLE-3 draft to Qwen3-30B-A3B loads cleanly — the load log shows the EAGLE-3 path engaging — and then the first generate() fails with Port for tensor name last_hidden_state was not found. Mechanism: the hidden-state transform (transform_hidden_state) pattern-matches only the dense FFN residual shape, matches zero nodes on a MoE graph, and silently no-ops (no warning or error at load). 4/4 MoE runs fail with the byte-identical error — two community draft heads (Tengyunw, AngelSlim), GPU and CPU, with and without OFFLOAD_RATIO (the offload property is exonerated) — while the dense positive control works on the same stack, export recipe, and invocation (Qwen3-1.7B + AngelSlim's 1.7B head: 64 tokens at 25.14 tok/s; get_num_accepted_tokens() = 27, and openvino-genai's own summary for that run prints AVG acceptance rate, %: 15.1714 and Generated tokens by draft model: 174). So "does EAGLE-3 compose with OFFLOAD_RATIO" is unanswerable on this model class today. Limits: one MoE model tested (the official int4 IR), dense control N=1 (fp16, GPU).

The 35B vision-language MoE: what offloading unlocks (2026-08-29)

This section also records why the Omni pipeline could not be measured — the VLM is the closest observable proxy. The headline result is the ratio curve further down.

openvino.genai 2026.3 also added OmniPipeline (preview; Qwen3-Omni text+speech, composing a VLMPipeline "thinker" + talker). Its constructor forwards device properties the same way as LLMPipeline (verified in py_omni_pipeline.cpp, releases/2026/3), so OFFLOAD_RATIO would reach the plugin — but as of 2026-08-29 no OpenVINO-format Qwen3-Omni model exists on Hugging Face (OpenVINO org 0/441; upstream ships transformers/GGUF/AWQ only), and the ctor rejects everything else (model_type == QWEN3_OMNI enforced). The blessed path is a manual optimum-cli export (the smallest upstream variant is a 70.5 GB download, HF API-verified); not attempted. Omni offloading is therefore unobservable today; this follow-up measures the closest observable proxy — VLMPipeline + `OFFLOAD_RATIO` on OpenVINO/Qwen3.6-35B-A3B-int4-ov (19.7 GB vision-language MoE, qwen3_5_moe, the very pipeline Omni would compose). All on driver 32.0.101.8826 (rollback from the 8991 window verified before any run), same harness discipline, 1068-token prompt (this model's tokenizer count of the same essay), 256 greedy tokens.

• —Forwarding proven for VLMPipeline at both levels: ctor kwargs reach the plugin (py_vlm_pipeline.cpp:298-300, releases/2026/3, matching the installed 2026.3.1.0 wheel), and behaviorally — GPU-shared steady state 17.73 GiB @ratio 0 vs 14.10 GiB @ratio 30 (3.63 GiB resident-set delta, short proof runs).
• —A third failure texture: steady paging at ratio 0. Fully resident, this 19.7 GB model decodes at 1.44 tok/s (N=3, sd 1.0%, TPOT ~693 ms) with committed peaking at 29.90 GiB against 31.3 GiB physical — sustained per-token paging, not the chaotic overcommit of the findings above. TTFT inverts (2.1 s resident vs ~22-23 s offloaded: the fixed cost is resident-expert upload, paid at load time when nothing streams).
• —Position-controlled ratio curve (authoritative). A drift protocol (below) found the sequential fine ordering untrustworthy, so ratios 30/50/25/20 were re-run interleaved (30,50,25,20 ×2, one fresh process per run, 60 s idle spacing); every ratio's early/late positions agree within 1%, licensing the ordering — 20: 21.4 > 25: 20.3 > 30: 18.6 > 50: 15.0 tok/s (gaps 5%/9%/25% all clear the <1% position spread). Peak measured: 21.4 tok/s at ratio 20 — 14.9× fully-resident — with committed still 5.65 GiB under physical. The paging boundary is bracketed only as (0, 20) — untested below 20; locating it needs finer steps and repeated runs. Greedy outputs byte-identical across ratios (sampled snippet).
• —Image arm (deterministic synthetic 1280×896 PNG, ratio 30, N=3): 13.86 tok/s median, TTFT 27.8 s; the pipeline reports 1166 effective input tokens vs 1068 text-only (+98 for the image at this resolution). Measured entirely inside the suppressed window (below): absolute values are lower bounds and the image-vs-text comparison is confounded.
• —No hang on the VLM path, 2/2 (A/B two-generates-on-one-pipeline checks, text and image at ratio 30) — consistent with the INT8-specific characterization above. New N=1 observation: warm-pipeline TTFT does not collapse on VLMPipeline (text 24.2→12.2 s, image 27.3→21.5 s) the way it does on LLMPipeline INT4 (0.25-1.6 s), and second-generate decode ran 14-16% slower than the first in both checks.
• —Drift methodology note. An end-of-session control showed no session drift (18.53 vs the morning 18.10 tok/s band at ratio 30), but per-run wall-clock analysis exposed a ~25% throughput-suppression window (12:40-12:52) opening right after the 15-minute ratio-0 paging config and outlasting a transient rebound — it contaminated the sequential ratio-50/25/20 and image numbers (memory was unaffected; sequential JSONLs are retained in vlm/ as a labeled case study). Mechanism uncharacterized — the harness logs no temperatures; thermal/power attribution is hypothesis. The interleaved table is the only fine-ordering source used here.

Study 1 — methodology

Stack and hardware. Core Ultra 7 258V (Lunar Lake) / Arc 140V (Xe2) integrated GPU, 32 GB LPDDR5X-8533 unified memory (31.3 GiB effective system pool), Windows 11 Pro build 26200. openvino 2026.3.1, openvino-genai 2026.3.1.0, openvino-tokenizers 2026.3.1.0 — pip wheels, no source build. GPU driver 32.0.101.8826 for every timing number in this study; driver 32.0.101.8991 was used only to re-validate the two failure modes, and those runs are labelled where they appear. The plugin's own GPU_DEVICE_TOTAL_MEM_SIZE reads 27,031,388,160 B (25.17 GiB) on this configuration — the full live SUPPORTED_PROPERTIES dump is published as gpu_supported_properties_2026.3.1.txt.

Models (HF revisions pinned; manifests published for download verification): OpenVINO/Qwen3-30B-A3B-int8-ov @ 2a8124554209bf5e5b4d75ca9aa37fb545c103a2 (30.8 GB), OpenVINO/Qwen3-30B-A3B-int4-ov @ 3fb72075ee93cdae480490bbac7a5e065172d0dd (16.3 GB), OpenVINO/Qwen3.6-35B-A3B-int4-ov @ 3a4443474bd304c5328abbe00054e1b145f6abae (19.7 GB, VLM).

Generation config. Greedy throughout (do_sample=False), ignore_eos=True, max_new_tokens=256 for every benchmark run. Prompts are published verbatim in reproducers/: the long prompt is 1054 tokens under the 30B tokenizer (1068 under the 35B VLM's), the short prompt 54 tokens, and the context probe 8,136 tokens; the VLM image arm reports 1166 effective input tokens for the same text plus a deterministic 1280×896 PNG.

One process per generate. Because of the INT8 second-generate hang (finding 1), every benchmark number in this study comes from a freshly launched process per generate — one warm-up run plus N measured runs, medians reported, N stated per row. That is a consequence of the bug, not a preference: it means every INT8 number pays the resident-expert upload cost on every generate, which is why TTFT here is not a warm-pipeline figure.

Memory accounting. System-wide counters sampled every 2–3 s (slower under load) by sampler.ps1: Windows committed bytes, and GPU Adapter Memory / Shared Usage summed across adapter instances. These are system-wide, not per-process accounting. Peaks and medians are both published, and memory_units_canonical_2026-08-29.csv gives every headline memory figure in bytes, decimal GB, and binary GiB side by side so the units cannot be confused.

Watchdog and gating. Each run is preceded by a memory gate (baseline committed / free physical, with retry and abandon) and wrapped by a watchdog that kills on silence and records the last memory samples alongside the kill — which is how the hang and thrash evidence was captured. Kill lines appear verbatim in status/.

Position-controlled runs (VLM). Sequential ordering proved untrustworthy after a ~25% throughput-suppression window was found mid-session, so the VLM fine ordering was re-measured interleaved (ratios 30, 50, 25, 20, twice through, one fresh process per run, 60 s idle spacing). Every ratio's early and late positions agree within 1%, and the interleaved table is the only fine-ordering source used. The contaminated sequential JSONLs are retained under vlm/ as a labelled case study rather than deleted.

Attention backend. All numbers come from GenAI's default PagedAttention (continuous-batching) backend, verified from compile logs by re-running at OPENVINO_LOG_LEVEL=3. An explicit ATTENTION_BACKEND=SDPA A/B on the VLM decoded 11.9% slower, so the default was also the faster backend here.

A labelled negative result. The ratio-60 long-prompt config was first measured while a concurrent 15 GB download shared the same NVMe; that run reads 3.79 tok/s against 4.68 for the clean re-run. Both are published — _fp (IO-contended) and _fpclean — and the CSV's notes column says which is which. The clean figure is the one quoted.

Reproducers. Every benchmark and diagnostic script is published verbatim under reproducers/ (and vlm/reproducers/, eagle3/reproducers/), including the minimal hang reproducer diag_second_gen.py and all chain orchestrators, together with the exact prompt files.

Study 1 — caveats and what I did not measure

• —Single machine, pip wheels, no source build. One unit, one OS, one thermal environment.
• —One driver for all timing. 32.0.101.8826; 32.0.101.8991 was exercised only to re-validate the two failure modes, not to re-measure throughput.
• —Output quality was not measured at any ratio. Greedy outputs were byte-identical across ratios on a sampled snippet; that is a consistency check, not a quality evaluation.
• —The hang was exercised at `OFFLOAD_RATIO` 75 and 60 only, and the prompt-length threshold between 54 and 1054 tokens is uncharacterised — a single boundary was not tested, and mixed prompt lengths (long first then short, or the reverse) were not exercised at all.
• —The VLM paging boundary is bracketed only as (0, 20). Nothing between was measured.
• —Memory figures are system-wide counters at 2–3 s resolution, not per-process accounting, and the sampler slows under load.
• —No discrete GPU was tested. On a dGPU the host-side streaming cache would spend otherwise-free host RAM; the unified-memory contention described here is specific to integrated GPUs.
• —No temperature or frequency logging accompanied the throughput runs, so the ~25% suppression window found in the VLM session is described but not attributed — thermal or power causation is hypothesis, not measurement.
• —EAGLE-3 addendum: one MoE model tested (the official INT4 IR); the dense positive control is N=1, fp16, GPU.
• —Mechanism language is inference, not instrumentation. Phrases like "consistent with a prefill/dispatch-phase hang" or "consistent with weight paging" describe what the black-box behaviour is consistent with; no GPU-plugin source was instrumented to confirm them.
• —GenAI `LLMPipeline` / `VLMPipeline` paths only — the raw core.compile_model path was not exercised.

What's in moe_offload_2026-08-28/

report_moe_offload_arc140v_2026-08-28.md (findings-first report), RUNLOG.md (full session timeline including dead ends), results/ (per-config JSONL, one record per generate with TTFT/TPOT/throughput/token counts from openvino_genai perfmetrics), `memory/` (committed + GPU Adapter Memory samples every 2-3 s — slower under load — per config), `status/` (append-only per-run status files incl. watchdog kills with memory evidence), `reproducers/` (all benchmark/diagnostic scripts verbatim, incl. the minimal hang reproducer `diagsecondgen.py`, plus the exact prompts), `traces/` (Intel Unified Telemetry v0.2.0-beta1.1 captures of the hang, the ratio-100 thrash, and a healthy ratio-60 generate, converted to Chrome-trace JSON with the l0gpu metric stream excluded for size; raw-bin manifest with byte sizes), gpu_supported_properties_2026.3.1.txt (live plugin property dump), and the HF file manifests used for download verification.

vlm/ (the 2026-08-29 follow-up, 67 files): results/ (13 per-config JSONLs — text arm at ratios 0/20/25/30/50, image arm, forwarding-proof runs, the end-of-session drift control, and the four interleaved position-controlled JSONLs with position labels pos1-pos8), memory/ (19 sampler CSVs incl. one per interleaved position — the standard-arm CSVs cover each config's final run window only), status/ (15 append-only status files incl. both A/B hang checks), reproducers/ (the VLM bench/diag scripts, all five chain orchestrators, the verify-then-launch script, the image prompt — the text prompts are the same files as in the parent reproducers/ — the deterministic 1280×896 test PNG, the model's HF manifest, and the stats/drift-table analysis scripts), plus the chain/verify/download logs at the folder root.

eagle3/ (the 2026-08-29 EAGLE-3 × MoE addendum, 36 files): status/ (7 append-only status files — the four MoE failures with verbatim tracebacks, the dense-control success, the same-session 30B baseline control, and the two-generate A/B check), pylogs/ (12 run logs, stdout+stderr pairs at OPENVINO_LOG_LEVEL=3 — the load-time log shows the EAGLE-3 path engaging on the MoE target, the embedding-copy INFO line plus main and draft both compiling as "LLM with Paged Attention", before generate() fails), results/ (2 JSONLs: the same-session 30B baseline control and the dense positive control with acceptance metrics), memory/ (6 sampler CSVs), reproducers/ (the EAGLE-3 bench/diag scripts and the chain orchestrator), export_logs/ (5 optimum-intel export logs, incl. the two failed first attempts and their resolution — --trust-remote-code routing plus the einops dependency), and the chain log at the folder root.

Study 2 — Model co-residency and the 2026.1.0 → 2026.2.1 delta (2026-08)

Stack: OpenVINO GenAI 2026.1.0 and 2026.2.1 (in-process) and OpenVINO Model Server 2026.2, GPU driver 32.0.101.8826. A different OpenVINO release from Study 1 — these numbers are not comparable with the offloading study above. Data in the top-level CSV/JSONL tables and `raw/`.

Key findings — co-residency and single-model

• —Single-model decode (median tok/s): 14B 11.1 (spec-off) → 17.1 (spec-on, ≈1.5×); 8B 19.8 → 27.4 (≈1.4×); 30B-A3B MoE 38.1. Spec-decode uses the small Qwen3-0.6B draft on the GPU. The MoE has the fastest decode and TTFT (213 ms) but the slowest prefill (≈480 tok/s vs ≈1960 for the dense 14B/8B) — it activates only ≈3B params per decode step but routes every prompt token through all experts at prefill. This dense-vs-MoE mirror-image is the most useful shape here.
• —Co-residency fit: all four pairings completed without out-of-memory, within both the 27.3 GiB iGPU window (the 87% override) and the 31.323 GiB system pool. The reported peaks (24.5–27.6 GiB) are total unified-pool system-RAM (the harness reads system-Available as a proxy) — an upper bound on the iGPU's own allocation, not the GPU slice alone — so the exact headroom against the 27.3 GiB window isn't isolated here; what is clear is that the 87% override is required (the default, smaller window would not hold a resident 14B plus a second model and its working set), and the heavy pairings push total use into the high-20s GiB (the style-LoRA case peaks at 27.6 GiB total, its tightest). The "Headroom vs pool" values elsewhere are measured against the 31.323 GiB total. Idle co-residence is effectively free for the SDXL partners (within run-to-run noise) and a consistent ≈5% for the VLM.
• —Co-residency under concurrent load: the resident 14B's decode is throttled to ≈1% of baseline by SDXL diffusion (compute-bound, monopolizes the execution-unit scheduler so completely the 14B can't finish even its prefill in the window), and to ≈13–17% by the bandwidth-bound VLM and the LoRA case. Across every pairing the GPU goes to ≈99% busy and stays pinned near 1.95 GHz with zero throttle — so the starvation is occupancy/compute-scheduling, not frequency, thermal, or bandwidth. The vector-engine counter supports this — XVE_ACTIVE is only ≈20–31% under contention (30.8 / 27.7 / 21.5 / 20.0% across the pairings) even at ≈99% GPU-busy, i.e. the GPU is occupied, not vector-compute-saturated. The memory-bandwidth signature splits by partner type: under compute-bound SDXL the aggregate read-rate drops (the 14B's bandwidth-bound decode can't get compute slots to issue its reads), while the bandwidth-bound VLM keeps it high.
• —NPU = 0 W in every phase of every run — a repeated confirmation that this is pure-GPU execution.

2026.2.1 update (OpenVINO GenAI 2026.1.0 → 2026.2.1)

A second campaign re-measured this stack across the OpenVINO GenAI 2026.1.0 → 2026.2.1 bump, holding hardware, GPU driver (32.0.101.8826), models, and methodology constant (the two versions were run back-to-back in separate Python venvs on the same machine; OVMS 2026.2 was unchanged, so the 30B section is a flag A/B, not a version A/B). The useful headline: the bump makes prompt processing meaningfully faster — cold prefill ≈28–36% at the tightest 512-token point and the vision model's TTFT ≈19% — while steady-state token generation is unchanged (so a tokens/sec-only look would read it as a no-op and miss the gain). Narrative write-up: https://github.com/openvinotoolkit/openvino/discussions/36604 (co-residency) and the 2026.2.1 follow-up discussion (https://github.com/openvinotoolkit/openvino/discussions/36626). New tables in this dataset: version_delta, draft_device, moe_flag, coresidency_2026_2_1.

Cold prefill (prompt processing) — faster on 2026.2.1

N=5, cold prefill (prefix-caching off), no draft. The 512-token points are the tight, solid claim (std drops to single digits on 2026.2.1); mid-range gains are larger but higher-variance.

Prefill: cold vs prefix-cache-warm (reconciliation). The single-model single_model table above reports ≈1960 tok/s for the dense 14B/8B — but that probe runs prefix-cache-warm (caching on, over a repeated prompt set), so it is an optimistic upper bound, not a cold rate. This version_delta harness is cold (prefix-caching off), which is why it reads ≈595–760 for the 14B at comparable length — a ≈2.5× gap explained entirely by prefix caching. For a cold / first-prompt or cross-runtime `pp512` comparison, use the cold numbers here (e.g. 14B 761, 8B 1477 on 2026.2.1); the ≈1960 figure only applies to repeated prompts that hit the KV cache. (This also means the dense-vs-MoE prefill gap is ≈1.3–1.6× cold, not the ≈4× a warm-vs-cold comparison would suggest — the MoE still has the slowest prefill of the set, just by a smaller cold margin.)

Vision model (Qwen3-VL-8B-Instruct INT4) — faster TTFT

TTFT (median) 201.3 → 161.8 ms (≈19.6% faster, the solid claim); TPOT 63.0 → 47.3 ms (directional — the 2026.1.0 TPOT sample was noisier). Fixed image + prompt.

Token generation — unchanged (stated plainly so the prefill/VLM win above isn't read as a generation speedup)

Median tok/s, greedy; flat both directions within run-to-run noise — a tokens/sec-only look would read the bump as a no-op.

Speculative-decoding draft device — keep the draft on the GPU

The CPU draft is ≈13% slower at identical acceptance — on a unified-memory iGPU it pulls the same LPDDR5X and adds a CPU↔GPU sync per round for no benefit. This completes the draft-device ordering with the earlier NPU-draft study (https://github.com/openvinotoolkit/openvino/discussions/36484): GPU > CPU > NPU (different draft builds/harnesses between the two studies — compare the ordering, not absolute tok/s).

30B MoE accuracy flag (MOE_USE_MICRO_GEMM_PREFILL=0) — a measured throughput cost

The flag costs ≈19% generation + ≈28% TTFT; the 2.2K-token eval found no accuracy difference (its documented benefit needs 8K–32K context — not demonstrated here, only the cost is measured).

Annotation added 2026-09-15 — a condition these two rows do not record. The arms above were served through OVMS, and `--cache_size` was not captured as a column for them. A later single-variable bisect on the same host, same model and same driver, with no offload involved, measured --cache_size alone moving decode throughput from 36.5–39.3 tok/s at 1 GB to 2.9–3.0 tok/s at 4 GB — roughly a 13x swing, an order of magnitude larger than the ~19% effect this table reports. What this does and does not change. The within-table comparison is probably unaffected: both arms were run back-to-back in one campaign and so almost certainly shared the setting. What is not safe is treating 38.58 tok/s as a portable absolute for this model on this hardware, or comparing it against any run whose --cache_size differed. A related contradiction is open and unexplained: the same nominal OVMS configuration measured 38.58 tok/s in June and 3.02 tok/s in August. Until that is resolved, treat OVMS throughput figures in this dataset as conditional. Direct-measured GPU figures elsewhere in this dataset do not run through OVMS and are not affected. Recorded here rather than silently corrected, because the numbers were published without the condition that governs them.

Second generate() hang vs OFFLOAD_RATIO (2026-09-14)

secondgen_36891_2026-09-14/ — 28 rows, per-unit counter samples and harness for a before/after comparison of openvino#36891 against the second-generate() hang in openvino#37736, on Arc 140V. Three builds (PR head, its merge-base, the 2026.3.1 wheel) at two offload ratios. See that folder's README.md for results, the columns that failed to discriminate, and the caveats.

Co-residency on 2026.2.1 (N=1 confirmatory refresh)

The co-residency pattern from the N=3 study (coresidency table / discussion #36604) holds on 2026.2.1: idle co-residence is ≈free, but concurrent generation saturates the iGPU and starves the 14B; GPU-busy → ≈100% while GPU memory-read rate falls (e.g. 66 → 19 GB/s under SDXL), core clock pinned ≈1.92 GHz, zero throttle, NPU 0 W — compute-scheduling-bound, not bandwidth or clock. N=1 here (a pattern-holds check); the full N=3 variance study is the coresidency table. Methodology note: the SDXL partners here are 20-step (vs 30-step in the N=3 study) — a runtime-config change between campaigns; it lengthens/shortens the partner's generation but does not affect the ≈1% retained-throughput finding (which is about compute monopolization, not diffusion duration). The SDXL partners are RealVisXL V5.0 (SG161222/RealVisXL_V5.0) and SDXL base 1.0 (stabilityai/stable-diffusion-xl-base-1.0) ± a style LoRA — identities confirmed from the converted models' model_index.json provenance. On the bandwidth unit (`GPU_MEMORY_BYTE_READ_RATE`): the capture tool leaves it N/A; my reading of Intel's OA metric definition (`units="gbps"`, equation = byte-counter ÷ GpuTime in ns) in Mesa's oa-lnl.xml indicates GB/s — gigaBYTES, not bits (no ×8) — so I treat it as GB/s; correction welcome if the OA spec intends otherwise.

New files in this 2026.2.1 extension

version_delta.{csv,jsonl}, draft_device.{csv,jsonl}, moe_flag.{csv,jsonl}, coresidency_2026_2_1.{csv,jsonl} (the tables above), plus the unaggregated 2026.2.1 per-run source JSONs under raw/ (the generation/prefill/VLM A/B runs, the draft-device and acceptance probes, the OVMS MoE flag arms + coding eval, the SDXL latency + min_p runs, and the four co-residency runs with their per-phase Intel-UT socwatch + Level-Zero metrics).

Study 2 — files

Study 2 — hardware and software

• —CPU / SoC: Intel Core Ultra 7 258V (Lunar Lake)
• —GPU: Intel Arc 140V (Xe2, integrated), shared LPDDR5X-8533, ≈136 GB/s theoretical memory bandwidth, 31.323 GiB effective shared budget
• —Shared GPU memory: the Intel Graphics Software "Shared GPU Memory" override is set to 87%, giving the iGPU a 27.3 GiB addressable window into the unified pool (the value shown in Intel Graphics Software and in Windows Task Manager → Performance → GPU). This is the enabling configuration for the co-residency results — the default, smaller window will not hold a resident 14B plus a second model, so reproducers must set the same override.
• —GPU driver: 32.0.101.8826
• —OS: Windows 11
• —Runtimes: OpenVINO GenAI 2026.1.0 (in-process, for the 14B and 8B); OpenVINO Model Server (OVMS) 2026.2 (for the 30B MoE, continuous batching)
• —Models: Qwen3-14B INT4, Qwen3-8B INT4, Qwen3-Coder-30B-A3B INT4 (MoE), Qwen3-VL-8B-Instruct INT4 (vision), and SDXL-class INT8 image models (RealVisXL V5.0 and SDXL base 1.0, ± a style LoRA). Speculative-decoding draft: Qwen3-0.6B (a pruned-6-layer INT8 build) on the GPU.

Study 2 — methodology

Single-model (`single_model`). Each model was benchmarked alone: 5 measured runs + 2 warm-up, a 30 s inter-run cooldown for thermal fairness, greedy decoding (temperature 0), over a fixed 4-prompt set (so the num_runs column is 5 runs × 4 prompts = 20 measured generations per model/config). Generation throughput = output tokens / total latency. TTFT = time-to-first-token (a latency). Prefill (pp) = input tokens / prefill time, via a fixed ≈450–970-token probe with max_new_tokens=1. The 14B/8B in-process numbers and the 30B OVMS numbers are on different OpenVINO versions (2026.1.0 vs 2026.2) — each row states its own runtime.

Co-residency (`coresidency`). A Qwen3-14B (spec-decode on) is held resident, then for each partner model the full 14B suite (gen tok/s, prefill, TTFT) is measured in three states: baseline (14B alone), idle (partner loaded but not generating), and contention (partner generating continuously while the 14B runs back-to-back for a fixed 15 s window — a sustained probe that removes the overlap-timing noise of a one-shot measurement). N=3 repeats per pairing; the tables report the mean (the raw/ files hold every run, and the per-row standard deviations are included for the headline throughput). The co-residency baseline of ≈19.8 tok/s comes from this sustained back-to-back probe and reads a little higher than the single-model spec-on median of 17.07 tok/s in the single_model table — a different probe; the retained-% figures are relative to this harness's own baseline.

Telemetry. Each co-residency run is wrapped in Intel Unified Telemetry (ut.exe): Intel SoC Watch for energy → power (averaged as total mJ / total ms; peaks taken over 1 s windows to avoid sub-millisecond glitches), SoC/CPU temperature, throttle reason, and NPU power; Level Zero for GPU core frequency, GPU-busy %, and memory read/write bandwidth. Samples are segmented into phases by wall-clock boundaries the harness emits.

Units note — memory bandwidth. The Level Zero / OA memory-bandwidth metric (GPU_MEMORY_BYTE_READ_RATE / _WRITE_RATE) is in GB/s (gigabytes/second). Intel's OA metric definition for Lunar Lake declares it units="gbps" and derives it as the raw byte counter (GPU_MEMORY_BYTE_READ, units="bytes") divided by GPU time — bytes per nanosecond, i.e. gigabytes/second. (Source: Intel's metric set as captured in Mesa's oa-lnl.xml.) The capture tool reports this unit as N/A, but Intel's definition is explicit; the ≈108 GB/s peak here is ≈79% of the 136 GB/s LPDDR5X-8533 ceiling.

What's hard to find elsewhere (Study 2)

Most "what can this laptop run" data is single-model tokens/second. The parts that are hard to find elsewhere:

1. 1.Model co-residency cost. What a resident Qwen3-14B (INT4, speculative decoding on) costs when a second model — SDXL image generation, or a Qwen3-VL-8B vision model — shares the same iGPU and the same shared system RAM. Every pairing runs within the iGPU's 27.3 GiB shared-memory window without out-of-memory (peak 24.5–27.6 GiB total unified-pool system-RAM), idle co-residence is near-free, but concurrent generation saturates the GPU, and the exhausted resource is compute scheduling — not memory bandwidth and not clock.
2. 2.Per-phase GPU telemetry via Intel Unified Telemetry (Intel SoC Watch + Level Zero): iGPU-rail power, GPU frequency, GPU-busy %, memory-bandwidth, and a 0 W NPU confirming pure-GPU execution — not just wall-clock tokens/second.
3. 3.A Mixture-of-Experts (MoE) prefill/decode profile — Qwen3-Coder-30B-A3B (≈3B active params) on this iGPU.
4. 4.Speculative-decoding numbers for the 14B and 8B (draft model on the GPU), spec-off vs spec-on.

Study 2 — caveats and what I did not measure

• —Single machine, N=3. One unit, one driver, one thermal environment. Not a fleet; not a distribution. Treat as a reference point, not a population estimate.
• —The cartoon pairing's contention figure is soft. Its milder slowdown partly reflects a style-LoRA CPU-side spin-up landing inside the fixed 15 s window (hence its higher variance). Read it as "the mildest SDXL pressure," not a steady-state diffusion number.
• —Level Zero per-phase timing is via a validated linear clock-remap. The driver flags a timestamp-units uncertainty; values are correct, fine-grained alignment is approximate.
• —Speculative-decoding prefill is a backend artifact. The spec-on prefill probe reads low because a single-token probe still pays draft+target overhead; prefill is config-independent, so cite the spec-off prefill — though note that probe runs prefix-cache-warm (repeated prompts hit the KV cache), so its ≈1960 tok/s is an upper bound, not a cold rate; the controlled cold prefill is ≈595–761 for the 14B (see the 2026.2.1 update for the cold/warm reconciliation).
• —Cross-model prefill is directional, not a controlled A/B. The in-process probe is over ≈970 formatted tokens; the OVMS probe over ≈421 bare tokens; the runtimes differ. Each is internally valid; the dense-vs-MoE prefill gap is far larger than those confounds.
• —No co-resident cost is in `single_model` — those models were each benchmarked alone (the 30B and the 14B cannot co-reside within the 27.3 GiB iGPU window).

Both studies

Feedback I'd especially value

I'm a non-expert sharing measurements, so methodology critique is genuinely wanted — please tell me where this is wrong or could be sharper.

On Study 1 (MoE offloading):

• —The ratio floor. I bracketed the VLM paging boundary only as (0, 20). Is there a better way to locate it than a finer sweep with repeated runs — something that reads the resident set directly rather than inferring it from throughput?
• —Memory accounting. I sample system-wide committed bytes and GPU Adapter Shared Usage every 2–3 s. Is there a per-process or per-allocation view on Windows that would let me attribute the offload cache's own footprint instead of reasoning from system totals?
• —The suppression window. A ~25% throughput dip opened mid-session and I could not attribute it — the harness logs no temperatures. What would you log to settle thermal vs power vs something else?
• —The fresh-process-per-generate workaround. It is forced by the hang, and it clearly changes TTFT. What else might it be distorting?

On Study 2 (co-residency and version delta):

• —The co-residency contention probe. I measure the resident model's retained throughput by running it back-to-back for a fixed 15 s window while the partner generates continuously. Is there a more standard or fairer way to quantify "what a second model costs the first" on one shared GPU?
• —Steady-state vs spin-up. The style-LoRA pairing's contention figure is partly a fixed-window spin-up artifact; I plan to re-run with the partner warmed to steady state first. Is that the right fix, or is there a cleaner approach?
• —Prefill comparability. My prefill probe uses different prompt lengths across models/runtimes. Is there a community-standard prefill probe (e.g. a fixed pp512) I should align to so these compare cleanly across runtimes?
• —Replication. This is one unit, one driver, N=3. If you have a Lunar Lake / Arc 140V machine and see different numbers, I'd really like to know.

What would be useful to measure next?

If any of these would be valuable to you, say so and I'll try to add them: longer-context decode curves (these are short prompts); tokens-per-joule (I have per-phase power); other model sizes or architectures on this iGPU; additional co-residency pairings; or batched/concurrent throughput. Requests and suggestions are welcome.

Reproduce

Study 1 (MoE offloading). Everything needed is published under moe_offload_2026-08-28/reproducers/ — the benchmark scripts, the chain orchestrators, the watchdog and memory sampler, and the exact prompt files, all verbatim as run. The minimal hang reproducer is diag_second_gen.py, and gpu_supported_properties_2026.3.1.txt records the plugin configuration the runs saw. Per-generate records are in results/ (one JSONL row per generate, with TTFT/TPOT/throughput/token counts straight from openvino_genai perf_metrics), memory samples in memory/, and the append-only run logs — including every watchdog kill with its memory evidence — in status/. The VLM and EAGLE-3 follow-ups carry their own reproducers/ under vlm/ and eagle3/.

Study 2 (co-residency). The Study 2 methodology section above describes the harnesses: a fixed-prompt-set single-model benchmark (generation plus a max_new_tokens=1 prefill probe), and a co-residency harness that loads the 14B once and measures it against each partner at baseline / idle / contention with Intel UT wrapping each run. The raw/ per-run files contain the exact configs, phase timestamps, and per-run numbers behind every aggregate.

AI assistance

This is an AI-assisted contribution. The benchmark measurements here are real, first-hand results — captured on the contributor's own hardware (the Intel Arc 140V described below) via OpenVINO, and reproducible from the methodology here. The analysis, the dataset packaging, and this card were drafted with an AI assistant working under the contributor's direction. The contributor (a non-expert) owns the goals, the hardware, and the decisions, has reviewed and verified every number and claim, and takes full responsibility for them. The numbers come from measurement runs, not from a model — but please weigh the analysis and framing with that in mind, and corrections are especially welcome.

License and citation

Released under CC-BY-4.0. If this is useful, attribution is appreciated:

OpenVINO local-inference on Intel Arc 140V (Lunar Lake) — GPU MoE expert offloading, and a 14B co-residency study. GPU driver 32.0.101.8826 throughout. Study 1 (MoE offloading, moe_offload_2026-08-28/): openvino 2026.3.1 / openvino-genai 2026.3.1.0. Study 2 (co-residency and version delta): OpenVINO GenAI 2026.1.0 and 2026.2.1 / OVMS 2026.2. Cite the stack belonging to the study the number came from — the two are different releases.

This is a single contributor's measurements on one laptop, shared in the spirit of the OpenVINO and local-inference community. Corrections, replications on other Lunar Lake units, and methodology critiques are all welcome.