xiaosuhu1986/DocTrek-LLM-cost-estimation

LLM Cost, Capacity, Latency & Batch Sizer

Tabs: 1) Cost & Capacity – Managed API vs GPU costs (busy-time vs scheduled uptime; set 24 h/day for always-on). 2) Latency Estimator – prefill + decode + overhead, scaled by Queue/Burst factor for p95. 3) Batch Size Calculator – computes theoretical & recommended safe batch from VRAM and KV-cache math.

KV cache rule: KV ≈ 2 × hidden_size × bytes/elem × layers × seq_len × batch_size

Use KV precision 4/8/16 bits, and reserve headroom to avoid OOMs.