vireshk/LLM-Inference-Optimization
LLM Inference Optimization — quantization on an NVIDIA A40
Latency-vs-throughput curves, quality, and cost for quantized Llama-3.1-8B serving on a bandwidth-bound GPU — measured open-loop, with the tail reported and the measurement's own validity checked. The methodology is the product; the numbers are its output.
244 measured runs across 6 configurations, 6 quality evaluations, open-loop, at least three repeats per point.
Under a p95 TTFT budget of 500 ms, vllm-int8-w8a8 is cost-optimal: 2116 output tokens/sec at $0.0525 per million tokens — 27 % cheaper than vllm-bf16. vllm-bf16 is dominated: 4 configurations are cheaper at equal-or-better measured quality.
The grey band on the chart is vllm-bf16's 95 % confidence interval on GSM8K, and every configuration falls inside it. The quality spread across all 6 configurations is 0.0159, smaller than a single configuration's 95 % half-width of 0.0240 — quality does not separate these configurations at this sample size, so the decision is made on cost and latency.
The interactive explorer is the app above: set a p95 TTFT budget and see which configuration is cheapest under it.
- Repository, methodology and full report: https://github.com/vireshkoli/LLM-Inference-Optimization
- Findings: https://github.com/vireshkoli/LLM-Inference-Optimization/blob/main/REPORT.md
- How it was measured, and which numbers are not trustworthy: https://github.com/vireshkoli/LLM-Inference-Optimization/blob/main/METHODOLOGY.md
Every number here is generated from results/*.json in the repository by make report; nothing is hand-typed. This Space is vireshk/LLM-Inference-Optimization, deployed by make deploy-hf.