Tsunamisingh/vllm-prefix-demo

Prefix-caching demo (simulated)

This Space demonstrates the idea of prefix-caching (prefill + KV reuse) for demonstration only — it does not run vLLM or real GPU KV caching. It simulates prefill cost in order to make the latency/cache benefit visually obvious — it is intentionally lightweight so anyone can run it in-browser without GPU.

What this demo does

• —Runs a small language model in the Space and simulates an expensive "prefill" step (via a sleep).
• —Shows stateless vs cached behavior:
• —stateless — every request pays the prefill cost
• —cached — first request populates an in-memory cache, subsequent requests reuse it (prefill ~ 0s)

What this demo does NOT do

• —It does not implement GPU KV caching (vLLM-style). Real KV tensors and vLLM runtime behavior require a GPU runtime and vLLM or similar serving stack.
• —The in-memory cache is ephemeral — restarting the Space or committing changes clears it.

How to try

1. 1.Open App in this Space.
2. 2.Set Mode to cached and simulated prefill to 1.5s.
3. 3.Ask a short question (e.g., “Explain photosynthesis in one sentence.”) — first run should show Cache: MISS and ~1.5s prefill.
4. 4.Press Generate again (same question) — expect Cache: HIT and prefill ≈ 0.00s.
5. 5.Use Clear cache to reset.

For infra teams / recruiters

• —This demo is conceptual and meant to communicate the UX and cost-value of prefix caching. For production benchmarks and p50/p95 measurements, run the provided Colab recipe that boots vLLM on a GPU and measures real KV reuse speedups.
• —Contact Maria S or open an issue in the repo to request a 15-minute walkthrough and the GPU benchmark runs.

Implementation notes

• —The cache is implemented as an in-memory Python dict keyed by the system prefix string.
• —There is an optional on-disk persistence stub in app.py (commented) for simple demos, but it is not recommended for production.

Quick test steps (what to expect)

1. 1.Set Mode = cached, prefill = 1.5s.

1. 1.Enter the same question (e.g. “Explain photosynthesis in one sentence.”) and click Generate.

Expected: Cache: MISS, Prefill ≈ 1.50s. (First request populates cache.)

1. 1.Click Generate again (same question).

Expected: Cache: HIT, Prefill ≈ 0.00s, Total ≈ Gen time only. (Cache being used.)

1. 1.Click Clear cache then Generate again.

Expected: Cache: MISS again with Prefill ≈ 1.50s.

If you observe those behaviors, the simulation and UI are functioning correctly.

Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference