cogeanu-marius/qwen3.6-27b-h100-bf16-benchmark

Two GPUs do not mean twice the users

This study answers a serving decision, not a hardware trivia question: when a 27B model already fits on one H100, should a second GPU shard the model or run a second independent replica?

The answer

Concurrency is a load-generator setting, not a user count and not a promise. Capacity is the highest real arrival rate that satisfies a declared service objective. That is why this study measures both a saturated concurrency curve and finite Poisson arrivals with a good-request definition.

The earlier one-run c16 p95 TTFT of 9.01 seconds was not a stable c16 result. Repeating each point three times moved the one-time graph/JIT cost into the first encounter with a new shape; the median one-GPU c16 result is 2.11 seconds. The previous page confused warm-up behavior with steady capacity.

Publication status: core result complete, capacity knee pending

This page publishes the completed two-GPU core study now rather than hiding the useful result until every extension is finished. The repeated c1-c32 saturated curve, finite arrivals from 0.5 through 3 requests/s, and the long-input checks are measured evidence. The higher-load boundary points below are scheduled work, not results and not implied by the current data.

Maximum tested concurrency by p95 TTFT target

These are workload-specific bounds for 1,024 input and 256 output tokens. They must not be extrapolated as 16 users per GPU.

At c32, TP2 produced 995.3 output tok/s, or 497.7 per GPU: 1.71x the one-GPU result and 85.6% scaling efficiency. Two replicas produced 735.4 output tok/s, or 367.7 per GPU: 1.26x and 63.2% efficiency.

Repeated saturated-queue results

Each cell below is the median of three complete runs. Each concurrency point has at least 16 requests and at least two full waves of work.

Production-shaped arrival-rate result

Good requests satisfy all three objectives: p95-style per-request thresholds of TTFT <=1,000 ms, TPOT <=50 ms, and end-to-end latency <=15,000 ms. The measured peak goodput was 2.00 requests/s for TP2 and 1.64 requests/s for two replicas. See the machine-readable results and benchmark files for every offered rate from 0.5 through 3 requests/s.

What an 8-GPU node is actually for

An 8-GPU node is justified when the model or context needs aggregate memory, when tensor/expert parallelism improves a measured latency target enough to pay for communication, or when several replicas are needed to absorb a measured arrival rate with headroom and fault tolerance. It is not justified by writing c16 x 8 = c128.

This result is useful to inference engineers, platform teams, and technical owners setting admission control, autoscaling thresholds, replica counts, and latency SLOs. Application owners provide the traffic distribution and service target; the benchmark maps those requirements to a deployment.

How other teams use large H100 pools

• —Hugging Face demonstrates a 235B-parameter FP8 MoE on one 8x H100 node with data plus expert parallelism, a model/topology reason rather than a synthetic user-count target: native-speed vLLM Transformers backend.
• —TNG describes a 24-H100 self-hosted cluster in terms of concurrent processing, prefill, decode, utilization, and queue behavior: prefill and decode for concurrent requests.
• —NVIDIA measured all-reduce communication as a material part of decode latency for an 8-way tensor-parallel model, showing why TP is not free: low-latency communication for inference.
• —Hugging Face Inference Endpoints scales replica counts from utilization or pending requests, which is how measured capacity becomes an operational policy: autoscaling guide.

Reproducibility and evidence

• —Model: Qwen/Qwen3.6-27B, BF16, immutable revision 6a9e13bd6fc8f0983b9b99948120bc37f49c13e9
• —Engine: vLLM 0.26.0
• —Context ceiling: 32,768 tokens
• —Deployments: TP1, TP2 with NCCL/PyNCCL collectives, and two independent TP1 servers behind a streaming round-robin proxy
• —Run source: 609ff350b9e61fa019a77e1459a818e5ad1288f2
• —2-GPU result
• —2-GPU manifest

• —Execution plan
• —Suite
• —Original one-GPU baseline

Limits

• —This is one model, one engine version, one prompt shape, and one GPU topology.
• —Three repeats describe run-to-run stability but are not a universal capacity guarantee.
• —The proxy uses simple round robin, not queue-depth-aware routing.
• —No 4x or 8x claim is made. Those counts should be tested only after the 2x result identifies which architecture has a reason to scale.
• —Cost comparisons need equivalent cloud configurations and are outside this result.

What to expect next

Dates are planning windows, not claims that measurements already exist. If a run is delayed or fails validation, it will remain marked pending here.

Glossary