vimalnakrani/stillwarm-bench-results

stillwarm-bench results dataset

Every measured row behind the stillwarm project: does saving/restoring a llama-server conversation's KV cache to disk actually work, when does it beat recomputing, and what silently breaks it?

Hardware/build (frozen): MacBook Pro, Apple M3 Max, 36 GB unified memory, macOS 26.5.1; llama.cpp release b9871 (ef2d770…), Release build, Metal; models pinned by SHA-256 (Llama-3.1-8B-Instruct Q4KM, Qwen2.5-7B-Instruct Q4KM, Gemma-3-4B-it Q4KM — the exact SHA-256 is in every CSV row's model_sha256 column).

Methodology summary

• —Frozen workloads: Frankenstein (public domain, Project Gutenberg #84) cut to token-exact 2K–64K prefixes with the pinned build's tokenizer; fixed question set; greedy decoding (temp 0, top_k 1, seed 42), flash-attention explicitly ON in every measured run; warmup 1 + N=5 reps.
• —One CSV row per measured rep (schema v1/v1.1, 77+1 columns): full config, client-stopwatch TTFT, the server's own timings object verbatim, prompt_n (reuse evidence), memory, probe verdicts, file sizes, git SHAs.
• —Every warm/restored row carries two independent checks: cold_equivalent probe (64-token byte-diff vs a cold recompute; probe_result) and a reuse assertion (prompt_n ≤ new_input + 8; reuse_assert). Determinism (restore-vs-restore) is a separate, repro-verified property.
• —resume_total_ms (v1.1) = restore time + follow-up TTFT — the headline metric.
• —Hash-chained provenance: evidence.jsonl (AuditWeave, SHA-256 chain; the chain proves the log wasn't altered after writing — it does not prove the measurements were taken correctly).

Honesty notes (read before quoting numbers)

• —Backfill rule: 45 early warm rows predate outcome stamping and were deterministically backfilled on 2026-07-05 as a pure function of their recorded probe_result × reuse_assert (no measurement changed). All 256 warm rows are fully stamped; the aggregate counts only fully-stamped rows.
• —Thermal state matters: sustained load drifts prefill throughput (measured 8K f16 cold: 473 tok/s cool vs 310 tok/s after ~1 h load; 32K cold TTFT 139–170 s within one rung; 114 s on a cool machine). The headline numbers come from `blockA2_supervised.csv` (same-session, interleaved cold/restore); the full ladder (blockA_*.csv) is a thermally mixed appendix. A measured curiosity: purged-page-cache restores on a cool machine beat page-cache-warm restores on a hot one (64K; dd contrast 21.2 vs 6.4 GB/s rules out page cache as the cause — thermal hypothesis, mechanism not isolated).
• —page_cache_state distinguishes warm_read (automated) from cold_read (supervised sudo purge protocol).

Methodology evidence (minimal excerpts backing claims not derivable from CSVs alone)

• —Divergence example (q4_0, 8K, question q1): the restored continuation and the same-config cold recompute first differ at generated position 47 — cold: "…The stranger is a noble and cultivated man, who is haunted by a dark past…" vs restored: "…The stranger, who is recovering from his ordeal, begins to open up to Walton" — both fluent. The hashes are in blockC_8k_q4_0.csv (probe_baseline_sha = cold, probe_restored_sha = restored); the restored hash equals the f16 baseline generation's hash (blockA_8k.csv / determinism rows), i.e. the cold q4 path is the outlier.
• —Determinism (restore-vs-restore) evidence in the CSVs: in blockA_4k.csv, the two independent divergent rows (prefix_reuse q3 rep3 and ram_parked q3 rep3) carry the identical gen_tokens_sha256 — two separate reuse paths reproduced each other exactly while differing from cold; re-runs reproduced row hashes byte-for-byte.
• —Qwen 32K rung absence: llama-server caps the slot context at the model's training context (32,768 for Qwen2.5); the frozen 32K document tokenizes to 32,830 Qwen tokens, so that cell is impossible by construction — blockE_* contains 16k as Qwen's long rung.
• —SWA + after-generation states: the exhibit CSV shows partial reuse (prompt_n ≈ 520). When the saved state additionally ends with sampled tokens that a new question does not extend, the required rollback is impossible for an SWA cache without --swa-full, escalating to full re-prefill (config-inspection finding; the stillwarm sidecar records save_point for exactly this reason).

Key files

Links

• —Harness + frozen workloads + figures: https://github.com/vimalnakrani08/stillwarm-bench
• —Tool (pip install stillwarm): https://github.com/vimalnakrani08/stillwarm · https://pypi.org/project/stillwarm/
• —Article: https://huggingface.co/blog/vimalnakrani/stillwarm-kv-cache-persistence · Demo Space: https://huggingface.co/spaces/vimalnakrani/stillwarm-demo

Author: Vimal Nakrani. License: CC-BY-4.0 (data and card). The underlying source text is public domain; no model weights are included.