vireshk/LLM-Inference-Optimization
0
1<!doctype html>2<html lang="en">3<head>4<meta charset="utf-8">5<meta name="viewport" content="width=device-width, initial-scale=1">6<title>LLM Inference Optimization — results explorer</title>7<meta name="description" content="Open-loop benchmark results for quantized Llama-3.1-8B serving on an NVIDIA A40.">8<script src="https://cdnjs.cloudflare.com/ajax/libs/plotly.js/2.35.0/plotly.min.js"9 onerror="window.__plotlyFailed = true"></script>10<style>11 :root {12 --bg: #fcfcfb; --surface: #ffffff; --text: #0b0b0b; --muted: #52514e;13 --grid: #e3e2df; --accent: #2a78d6; --good: #1baf7a; --bad: #e34948;14 }15 @media (prefers-color-scheme: dark) {16 :root:not([data-theme="light"]) {17 --bg: #14140f; --surface: #1c1c18; --text: #f5f4ef; --muted: #a8a69e;18 --grid: #33322c; --accent: #6aa9ee; --good: #4fd0a0; --bad: #f08886;19 }20 }21 * { box-sizing: border-box; }22 body { margin: 0; background: var(--bg); color: var(--text);23 font: 15px/1.6 -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif; }24 .wrap { max-width: 1080px; margin: 0 auto; padding: 40px 22px 72px; }25 h1 { font-size: 28px; line-height: 1.25; margin: 0 0 8px; letter-spacing: -0.01em; }26 h2 { font-size: 18px; margin: 40px 0 10px; letter-spacing: -0.005em; }27 .lede { color: var(--muted); margin: 0 0 6px; max-width: 68ch; }28 .links { margin: 18px 0 0; font-size: 14px; }29 .links a { color: var(--accent); text-decoration: none; margin-right: 18px; }30 .links a:hover { text-decoration: underline; }31 .panel { background: var(--surface); border: 1px solid var(--grid);32 border-radius: 10px; padding: 20px; margin-top: 18px; }33 .sla-row { display: flex; align-items: center; gap: 16px; flex-wrap: wrap; }34 .sla-row label { font-weight: 600; white-space: nowrap; }35 input[type=range] { flex: 1 1 260px; min-width: 200px; accent-color: var(--accent); }36 .budget { font-variant-numeric: tabular-nums; font-weight: 600; min-width: 5.5em; }37 .verdict { margin-top: 16px; font-size: 15px; line-height: 1.7; }38 .verdict strong { font-size: 17px; }39 code { font-family: ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 0.92em;40 background: var(--bg); border: 1px solid var(--grid); border-radius: 4px; padding: 1px 5px; }41 .scroll { overflow-x: auto; -webkit-overflow-scrolling: touch; }42 table { border-collapse: collapse; width: 100%; font-size: 14px; min-width: 640px; }43 th, td { text-align: right; padding: 7px 10px; border-bottom: 1px solid var(--grid);44 font-variant-numeric: tabular-nums; white-space: nowrap; }45 th:first-child, td:first-child { text-align: left; font-variant-numeric: normal; }46 th { color: var(--muted); font-weight: 600; font-size: 12.5px; text-transform: uppercase;47 letter-spacing: 0.04em; }48 tr.out td { opacity: 0.32; }49 tr.best td { background: color-mix(in srgb, var(--good) 12%, transparent); font-weight: 600; }50 .tag { display: inline-block; font-size: 11.5px; padding: 1px 7px; border-radius: 999px;51 border: 1px solid currentColor; }52 .tag.in { color: var(--good); } .tag.out { color: var(--muted); }53 .note { color: var(--muted); font-size: 13.5px; margin-top: 12px; max-width: 74ch; }54 #chart { width: 100%; height: 460px; }55 footer { margin-top: 48px; padding-top: 20px; border-top: 1px solid var(--grid);56 color: var(--muted); font-size: 13.5px; }57</style>58</head>59<body>60<div class="wrap">61 62 <h1>Quantized LLM serving on an NVIDIA A40</h1>63 <p class="lede" id="lede">64 Llama-3.1-8B-Instruct · open-loop Poisson arrivals, at least 3 repeats per point,65 clocks locked, kernels asserted from engine logs.66 </p>67 <p class="lede">68 Every point below is a rate a server could actually be provisioned for. Oversubscribed69 runs — where offered load exceeded capacity and latency reflects run duration rather70 than load — are excluded here and published in the repository.71 </p>72 <p class="links">73 <a href="https://github.com/vireshkoli/LLM-Inference-Optimization">Repository</a>74 <a href="https://github.com/vireshkoli/LLM-Inference-Optimization/blob/main/REPORT.md">Findings</a>75 <a href="https://github.com/vireshkoli/LLM-Inference-Optimization/blob/main/METHODOLOGY.md">Methodology</a>76 </p>77 78 <h2>Pick a latency budget</h2>79 <div class="panel">80 <div class="sla-row">81 <label for="sla">p95 TTFT budget</label>82 <input type="range" id="sla" min="100" max="3000" step="25" value="500">83 <span class="budget" id="slaVal">500 ms</span>84 </div>85 <div class="verdict" id="verdict"></div>86 <p class="note">87 Because every configuration ran on the same GPU, the hourly price is a linear scalar and88 cannot reorder this ranking. The latency budget can, and does — it decides which89 offered rates are admissible, which caps sustainable throughput, which sets cost per token.90 </p>91 </div>92 93 <h2>Latency vs throughput</h2>94 <div class="panel">95 <div id="chart"></div>96 <p class="note">97 Click a legend entry to hide a configuration. The dashed line is the current budget;98 points above it do not meet the SLA. Log scale on the y axis, because tail latency spans99 orders of magnitude between the linear region and saturation.100 </p>101 </div>102 103 <h2>Operating points</h2>104 <div class="panel">105 <div class="scroll"><table id="tbl">106 <thead><tr>107 <th>Config</th><th>Rate</th><th>Throughput</th><th>TTFT p95</th>108 <th>TPOT p95</th><th>$/1M tokens</th><th>Repeats</th><th>SLA</th>109 </tr></thead>110 <tbody></tbody>111 </table></div>112 <p class="note">113 Throughput is mean across repeats. Cost assumes $0.40/GPU-hour (RunPod A40 community114 cloud, accessed 2026-08-10) — an assumption, stated as one, because the measurement115 GPU is a lab machine with no invoice.116 </p>117 </div>118 119 <footer>120 Generated from <code>results/runs/*.json</code> by <code>make report</code>. Nothing on this121 page is hand-typed. Full methodology, known confounds and limitations are in the repository.122 </footer>123</div>124 125<script>126// Fixed palette assigned by configuration identity, never by rank, so hiding a127// series never repaints the survivors.128const COLORS = ["#2a78d6","#eb6834","#1baf7a","#eda100","#e87ba4","#008300","#4a3aa7","#e34948"];129const dark = () => window.matchMedia("(prefers-color-scheme: dark)").matches;130 131let ROWS = [], CONFIGS = [], COLOR = {};132 133const fmtUsd = v => "$" + v.toFixed(4);134const fmtMs = v => v >= 1000 ? (v/1000).toFixed(2) + " s" : v.toFixed(0) + " ms";135 136function admissible(budget) {137 // Best point per config under the budget: highest throughput that still138 // meets it. Throughput is maximised rather than latency minimised because139 // for a fixed SLA the cheapest config is the one serving the most tokens140 // per GPU-hour while still inside it.141 const best = {};142 for (const r of ROWS) {143 if (r.ttft_p95_ms > budget) continue;144 if (!best[r.config_id] || r.throughput_tokens_s > best[r.config_id].throughput_tokens_s) {145 best[r.config_id] = r;146 }147 }148 return best;149}150 151function renderVerdict(budget) {152 const best = admissible(budget);153 const picks = Object.values(best);154 const el = document.getElementById("verdict");155 if (!picks.length) {156 el.innerHTML = "<strong>No configuration meets this budget.</strong> " +157 "The tightest measured p95 TTFT is " +158 fmtMs(Math.min(...ROWS.map(r => r.ttft_p95_ms))) + ".";159 return;160 }161 picks.sort((a, b) => a.cost_per_1m_usd - b.cost_per_1m_usd);162 const w = picks[0];163 const bf = best["vllm-bf16"];164 let vs = "";165 if (bf && bf.config_id !== w.config_id) {166 const saving = (1 - w.cost_per_1m_usd / bf.cost_per_1m_usd) * 100;167 vs = ` — ${saving.toFixed(0)}% cheaper than BF16, which serves ` +168 `${bf.throughput_tokens_s.toFixed(0)} tok/s at ${fmtUsd(bf.cost_per_1m_usd)}.`;169 }170 el.innerHTML =171 `Cheapest configuration meeting <strong>${budget} ms</strong>: ` +172 `<strong><code>${w.config_id}</code></strong> at ${w.rate_rps} req/s, ` +173 `serving <strong>${w.throughput_tokens_s.toFixed(0)} tok/s</strong> for ` +174 `<strong>${fmtUsd(w.cost_per_1m_usd)}</strong> per million output tokens${vs}` +175 `<br><span style="color:var(--muted)">${picks.length} of ${CONFIGS.length} ` +176 `configurations admissible at this budget.</span>`;177}178 179function renderTable(budget) {180 const best = admissible(budget);181 const cheapest = Object.values(best).sort((a,b) => a.cost_per_1m_usd - b.cost_per_1m_usd)[0];182 const body = document.getElementById("tbl").querySelector("tbody");183 body.innerHTML = ROWS.map(r => {184 const ok = r.ttft_p95_ms <= budget;185 const isBest = cheapest && r.config_id === cheapest.config_id && r.rate_rps === cheapest.rate_rps;186 return `<tr class="${isBest ? "best" : (ok ? "" : "out")}">187 <td><code>${r.config_id}</code></td>188 <td>${r.rate_rps} rps</td>189 <td>${r.throughput_tokens_s.toFixed(0)} ± ${r.throughput_std.toFixed(0)}</td>190 <td>${fmtMs(r.ttft_p95_ms)}</td>191 <td>${r.tpot_p95_ms.toFixed(1)} ms</td>192 <td>${fmtUsd(r.cost_per_1m_usd)}</td>193 <td>${r.repeats}</td>194 <td><span class="tag ${ok ? "in" : "out"}">${ok ? "meets" : "over"}</span></td>195 </tr>`;196 }).join("");197}198 199function renderChart(budget) {200 // Fail in the open. The chart library is the page's one external201 // dependency; if it does not load, an empty panel looks like a page with202 // no data, which is worse than a page that says what is wrong.203 if (typeof Plotly === "undefined") {204 document.getElementById("chart").innerHTML =205 '<p class="note" style="padding:32px 0;text-align:center">' +206 "The chart library failed to load from its CDN. The table and the budget " +207 "verdict above are computed locally and remain correct.</p>";208 return;209 }210 const muted = dark() ? "#a8a69e" : "#52514e";211 const grid = dark() ? "#33322c" : "#e3e2df";212 const traces = CONFIGS.map(cfg => {213 const pts = ROWS.filter(r => r.config_id === cfg)214 .sort((a,b) => a.throughput_tokens_s - b.throughput_tokens_s);215 return {216 x: pts.map(p => p.throughput_tokens_s),217 y: pts.map(p => p.ttft_p95_ms),218 name: cfg,219 type: "scatter",220 mode: "lines+markers",221 line: { color: COLOR[cfg], width: 2 },222 marker: { size: 7, color: COLOR[cfg] },223 hovertemplate:224 "<b>%{fullData.name}</b><br>%{x:.0f} tok/s<br>TTFT p95 %{y:.0f} ms" +225 "<extra></extra>"226 };227 });228 const layout = {229 margin: { l: 62, r: 18, t: 10, b: 52 },230 paper_bgcolor: "rgba(0,0,0,0)",231 plot_bgcolor: "rgba(0,0,0,0)",232 font: { color: muted, size: 12 },233 xaxis: { title: "Output token throughput (tokens/sec)", gridcolor: grid, zeroline: false },234 yaxis: { title: "p95 time to first token (ms)", type: "log", gridcolor: grid, zeroline: false },235 legend: { orientation: "h", y: -0.22, font: { size: 11.5 } },236 shapes: [{237 type: "line", xref: "paper", x0: 0, x1: 1, y0: budget, y1: budget,238 line: { color: muted, width: 1.4, dash: "dash" }239 }],240 annotations: [{241 xref: "paper", x: 1, y: Math.log10(budget), xanchor: "right", yanchor: "bottom",242 text: `SLA ${budget} ms`, showarrow: false, font: { size: 11, color: muted }243 }]244 };245 Plotly.react("chart", traces, layout, { displayModeBar: false, responsive: true });246}247 248function refresh() {249 const budget = Number(document.getElementById("sla").value);250 document.getElementById("slaVal").textContent = budget + " ms";251 renderVerdict(budget);252 renderTable(budget);253 renderChart(budget);254}255 256fetch("results.json")257 .then(r => r.json())258 .then(data => {259 ROWS = data;260 CONFIGS = [...new Set(data.map(r => r.config_id))].sort();261 CONFIGS.forEach((c, i) => { COLOR[c] = COLORS[i % COLORS.length]; });262 // Counts come from the data the page loaded, never from typed text: a263 // typed "168 measurements" survived two releases after it stopped being true.264 const repeats = data.reduce((n, r) => n + r.repeats, 0);265 document.getElementById("lede").innerHTML =266 `${data.length} reportable operating points from ${repeats} runs · ` +267 `${CONFIGS.length} configurations · Llama-3.1-8B-Instruct · ` +268 "open-loop Poisson arrivals, clocks locked, kernels asserted from engine logs.";269 document.getElementById("sla").addEventListener("input", refresh);270 window.matchMedia("(prefers-color-scheme: dark)")271 .addEventListener("change", () => renderChart(Number(document.getElementById("sla").value)));272 refresh();273 })274 .catch(err => {275 document.getElementById("verdict").textContent =276 "Could not load results.json: " + err.message;277 });278</script>279</body>280</html>281