Datasetpublic

Inferencebench/pass-at-k-benchmark-results

ISO-Bench Pass@k GPU Benchmark Results Agent-generated optimization patches benchmarked on real GPU hardware (NVIDIA H100 80GB). Scope: Lossfunk/ISO-Bench — 54 tasks (39 vLLM + 15 SGLang) Patches from: Inferencebench/pass-at-k-samples Summary vLLM SGLang Total Tasks benchmarked 29/39 10/15 39/54 Successful benchmarks 444 150 594 Agent vLLM SGLang Total Claude Code (Sonnet 4.5) 230 76 306 Codex CLI (GPT-5) 214 74 288… See the full description on the dataset page: https://huggingface.co/datasets/Inferencebench/pass-at-k-benchmark-results.

sourceHugging Faceupdated 6mo agoView on Hugging Face

0likes13downloads

Dataset Card

ISO-Bench Pass@k GPU Benchmark Results

Agent-generated optimization patches benchmarked on real GPU hardware (NVIDIA H100 80GB).

Scope: Lossfunk/ISO-Bench — 54 tasks (39 vLLM + 15 SGLang) Patches from: Inferencebench/pass-at-k-samples

Summary

	vLLM	SGLang	Total
Tasks benchmarked	29/39	10/15	39/54
Successful benchmarks	444	150	594

Agent	vLLM	SGLang	Total
Claude Code (Sonnet 4.5)	230	76	306
Codex CLI (GPT-5)	214	74	288

Schema

Column	Type	Description
`item_id`	str	ISO-Bench task ID (e.g., `vllm_core-0000`)
`sample_index`	int	Pass@k sample index (0-7)
`repo`	str	`vllm` or `sglang`
`agent_name`	str	`claude_code` or `codex_cli`
`agent_model`	str	`sonnet-4.5` or `gpt-5`
`human_commit`	str	Human optimization commit hash
`parent_commit`	str	Baseline (pre-optimization) commit hash
`benchmark_mode`	str	`serving`, `standalone`, `prefix_caching`, `offline`
`llm_model`	str	LLM model benchmarked (e.g., `meta-llama/Meta-Llama-3-8B-Instruct`)
`duration_s`	float	Total benchmark duration (seconds)

Serving Metrics

ttft_mean_ms, ttft_median_ms, ttft_p99_ms — Time To First Token tpot_mean_ms, tpot_median_ms, tpot_p99_ms — Time Per Output Token itl_mean_ms, itl_median_ms, itl_p99_ms — Inter-Token Latency request_throughput_req_s — Requests/sec output_token_throughput_tok_s — Output tokens/sec total_token_throughput_tok_s — Total tokens/sec

Latency Metrics

latency_avg_ms, latency_p50_ms, latency_p99_ms

Throughput Metrics

throughput_tok_s, elapsed_time_s, input_throughput_tok_s

Usage

python

from datasets import load_dataset

ds = load_dataset("Inferencebench/pass-at-k-benchmark-results", split="train")

# vLLM Claude Code results
vllm_cc = ds.filter(lambda x: x["repo"] == "vllm" and x["agent_name"] == "claude_code")

# SGLang Codex results
sg_cx = ds.filter(lambda x: x["repo"] == "sglang" and x["agent_name"] == "codex_cli")

# Compute pass@k for a specific task
task = ds.filter(lambda x: x["item_id"] == "vllm_core-0005")

Hardware & Method

—GPU: NVIDIA H100 80GB PCIe
—vLLM: Docker containers from shikhar481/vllm_fixed_human_images (baseline images)
—SGLang: Docker containers from ayushnangia16/nvidia-sglang-docker
—Execution: Persistent container per task, sequential samples, setup shared across agents
—Date: 2026-03-29 to 2026-03-31