josefchen/physical-ai-inference-gap

The Inverted Ladder

An interactive, editorial companion to "Memory-Bound but Not Bandwidth-Limited: The Physical AI Inference Gap in Batch-1 LLM Decode" (Josef Chen, KAIKAKU).

Light, paper-style layout with four tabs:

• —Intro — the workload and the headline inversion, plus a live console (pick model / GPU / context / rate → step time, analytic floor, R_floor, $/Mtok) and the animated bandwidth-flow river (useful vs wasted bandwidth per GPU).
• —Mechanism — the two clocks (memory cost vs above-floor overhead, the slower one binds), the empty-highway kernel timeline with a CUDA Graphs toggle, and the dexterous barrier with a draggable control-rate target.
• —Methods — the full 44-cell R_floor heatmap, a pre-registered falsification panel (1.15× kill line, H100 1.259× CI [1.253, 1.267], L4 null 1.028×), the L4 quantisation matrix, the H100 attention-backend matrix, the cost inversion, and the measurement protocol.
• —Sandbox — type a sentence and stream it token-by-token at each GPU's measured ms/token in real time. H100 + CUDA Graphs vs L4 + ExLlamaV2 vs L4 bf16.

How it works

Static Space: a single self-contained index.html with all measurement data inlined. No backend, no build step, no cold start. SVG/Canvas visualisations only; one font import (Google Fonts).

Data provenance

Every number is taken from the paper's released JSON artefacts. Step times are medians of 30 measured single-token decode steps after 5 warmup, batch 1, bf16, sdpa, on Modal cloud hosts. Rfloor is recomputed from first principles (`W` from `weightbytes, K from 2·nlayers·nkvheads·headdim·ctx·2` bytes) and matches the paper's tables to three decimals. Peak HBM: H100 3350, A100-80GB 2039, L40S 864, L4 300 GB/s.

The N=10 CUDA Graphs A/B on H100 Qwen-2.5-7B ctx 2048: 14.83 ms eager → 11.78 ms graphed, 1.259×, 95% bootstrap CI [1.253, 1.267].

Notes

Only the H100 ($3.50/hr) and L4 ($0.30/hr) rate defaults are paper-sourced (Modal, May 2026); A100/L40S are editable placeholders. The cost figure is step_time × rate; it excludes idle, networking, storage and batching. The highway's 10-kernel split is schematic (the active fraction equals the measured R_floor). Not affiliated with NVIDIA; model names are trademarks of their respective owners.

Add the arXiv link in index.html (ARXIV_URL) once the paper URL is confirmed.