vireshk/LLM-Inference-Optimization
0
1---2title: LLM Inference Optimization3emoji: 📊4colorFrom: blue5colorTo: green6sdk: static7pinned: false8license: mit9short_description: Quantized Llama-3.1-8B serving on an A40, measured open-loop10---11 12# LLM Inference Optimization — quantization on an NVIDIA A4013 14Latency-vs-throughput curves, quality, and cost for quantized Llama-3.1-8B serving on a15bandwidth-bound GPU — measured open-loop, with the tail reported and the measurement's own16validity checked. **The methodology is the product; the numbers are its output.**17 18**244 measured runs** across 6 configurations, 6 quality evaluations, open-loop, at least three repeats per point.19 20Under a **p95 TTFT budget of 500 ms**, `vllm-int8-w8a8` is cost-optimal: **2116 output tokens/sec at $0.0525 per million tokens** — 27 % cheaper than `vllm-bf16`. `vllm-bf16` is dominated: 4 configurations are cheaper at equal-or-better measured quality.21 22The grey band on the chart is `vllm-bf16`'s 95 % confidence interval on GSM8K, and every configuration falls inside it. The quality spread across all 6 configurations is 0.0159, smaller than a single configuration's 95 % half-width of 0.0240 — **quality does not separate these configurations at this sample size**, so the decision is made on cost and latency.23 2425 26**The interactive explorer is the app above**: set a p95 TTFT budget and see which27configuration is cheapest under it.28 29- **Repository, methodology and full report:** https://github.com/vireshkoli/LLM-Inference-Optimization30- **Findings:** https://github.com/vireshkoli/LLM-Inference-Optimization/blob/main/REPORT.md31- **How it was measured, and which numbers are not trustworthy:** https://github.com/vireshkoli/LLM-Inference-Optimization/blob/main/METHODOLOGY.md32 33Every number here is generated from `results/*.json` in the repository by `make report`;34nothing is hand-typed. This Space is `vireshk/LLM-Inference-Optimization`, deployed by `make deploy-hf`.35