NagaYu/molt-benchmark-results

Molt · elastic on-device inference measurements

Everything measured while building **Molt**, a runtime that moves a running generation onto a smaller model between two tokens, carrying the KV cache across, so an on-device LLM under memory pressure is neither reclaimed by the OS nor restarted from the prompt.

Published so the claims can be checked rather than taken on trust. The figures in the repo README and the results page are generated from these files; nothing is transcribed by hand.

Setup

Qwen2.5 ladder on CPU (float32), five prompts per condition, the same memory-pressure trace for every arm:

The trace drops the budget to 2 002 MiB in two steps, timed to land mid-answer.

Configs

conditions — one row per experimental arm

The headline table. A is always-large, B always-small, C restart-on-pressure, D Molt; D-* are ablations.

cost_sweep — transplant vs re-prefill by context length

Both rungs kept warm, so this isolates the KV work from model loading (which both strategies pay identically). 4.4–5.1× faster, 74–79% of the FLOPs avoided, 128→1 024 tokens.

token_series — per-token latency, 1 040 rows

One row per generated token: inter-token latency, which rung produced it, the memory pressure at that moment, and the resident footprint. This is the raw data behind the hero figure, and it is the file to look at if you want to check that the "no long stall" story holds token by token.

projector_residuals — how well the learned maps fit

Held-out relative RMS error per route, with the training residual alongside. Reported this way because a dense 257×128 map fitted on a few hundred tokens drives the training number to ~0.01 while generalising not at all.

Also included

• —results/*.json — the full benchmark output, including every event log
• —pressure_traces/*.json — the memory-pressure traces (replayable)
• —demo_session.json — the raw SSE stream of a live service session, timestamps and migration costs included. figures/demo.gif is a replay of exactly this.
• —figures/*.png — every figure, regenerable from results/

Three findings that argue against the design

Included because an ablation table that only confirms its own design is not an ablation table.

1. 1.Top-k layer recompute did not earn its cost. Recomputing the destination's last 6 of 24 layers natively bought a 2% smoother seam for 31× the switch cost (900 ms against 29 ms). On this ladder, turn it off.
2. 2.A naive truncated map beats the learned one on the seam metric (handoff JSD 0.192 vs 0.220) while producing perplexity 24.1 against 3.1. JSD is bounded and rewards blurring: the naive map induces attention 11.4× flatter than the destination model's own. Two metrics, read together, or you get fooled.
3. 3.Step-to-step JSD is useless as a continuity metric on real text. It saturates at ln 2 ≈ 0.693 (measured steady state 0.65–0.69), so a migration's contribution sits below the floor. It is retained in the JSON only so the reader can see why it was rejected.

Reproduce

git clone https://github.com/NagaYu/molt && cd molt
pip install -r requirements.txt
python scripts/train_projectors.py --ladder qwen
python benchmarks/run.py --ladder qwen --conditions all --cost-sweep --qos
python figures/make_figures.py

The hermetic test-suite (pytest, 72 tests, ~1 min) needs no downloads at all — it runs the same code paths on randomly-initialised tiny models.

Caveats

• —Single machine, single run per condition; no error bars. The kill counts are categorical and robust; the millisecond figures are one sample each.
• —CPU only. The portable INT8 path dequantises on use (bitsandbytes is CUDA-only), so tier1 buys memory and costs ~4× decode speed.
• —Memory pressure is a replayed trace by default, for reproducibility. A real allocating child process is available (--real-hog) and gives the same qualitative result, noisier.

Model (the fitted maps): `NagaYu/molt-kv-projectors-qwen2.5` · Demo: `NagaYu/molt` · Code: github.com/NagaYu/molt