Datasetpublic

ssakethch/h200-quantization-benchmarks

H200 Quantization Benchmarks Benchmark results for 40 quantized and non-quantized instruction-tuned LLMs evaluated on an NVIDIA H200 MIG (Multi-Instance GPU) setup. This dataset supports reproducible comparison of quantization methods (AWQ, GPTQ, fp8, bf16) across accuracy and throughput dimensions. Dataset Configs Config Description Rows accuracy Per-task accuracy results from lm-eval across all models ~240 accuracy_leaderboard Aggregated accuracy… See the full description on the dataset page: https://huggingface.co/datasets/ssakethch/h200-quantization-benchmarks.

sourceHugging Faceupdated 5mo agoView on Hugging Face

1likes17downloads

Dataset Card

H200 Quantization Benchmarks

Benchmark results for 40 quantized and non-quantized instruction-tuned LLMs evaluated on an NVIDIA H200 MIG (Multi-Instance GPU) setup. This dataset supports reproducible comparison of quantization methods (AWQ, GPTQ, fp8, bf16) across accuracy and throughput dimensions.

Dataset Configs

Config	Description	Rows
`accuracy`	Per-task accuracy results from lm-eval across all models	~240
`accuracy_leaderboard`	Aggregated accuracy scores per model for leaderboard ranking	—
`model_inventory`	Registry of all evaluated models with quantization metadata	—
`throughput`	Token throughput measurements (tokens/sec) via vLLM	—
`throughput_leaderboard`	Aggregated throughput scores per model for leaderboard ranking	—

Hardware

—GPU: NVIDIA H200 SXM (141 GB HBM3e)
—Setup: MIG (Multi-Instance GPU) partitioned instances
—Inference engine: vLLM

Quantization Methods

Method	Description
`awq`	Activation-aware Weight Quantization (INT4)
`gptq`	GPTQ post-training quantization (INT4)
`fp8`	8-bit floating point (native H200 support)
`bf16`	BFloat16 baseline (no quantization)

Evaluation Tasks

Accuracy benchmarks are run with lm-evaluation-harness across:

—arc_challenge — ARC Challenge (25-shot)
—hellaswag — HellaSwag (10-shot)
—winogrande — Winogrande (5-shot)
—gsm8k — GSM8K math reasoning (5-shot)
—mmlu / mmlu_abstract_algebra — MMLU (5-shot)

Usage

python

from datasets import load_dataset

# Load accuracy benchmarks
accuracy = load_dataset("ssakethch/h200-quantization-benchmarks", "accuracy")

# Load model inventory
inventory = load_dataset("ssakethch/h200-quantization-benchmarks", "model_inventory")

# Load throughput benchmarks
throughput = load_dataset("ssakethch/h200-quantization-benchmarks", "throughput")

# Load leaderboards
acc_lb = load_dataset("ssakethch/h200-quantization-benchmarks", "accuracy_leaderboard")
thr_lb = load_dataset("ssakethch/h200-quantization-benchmarks", "throughput_leaderboard")

Schema

`accuracy`

Column	Type	Description
`timestamp`	string	Unix timestamp of the evaluation run
`model`	string	Full HuggingFace model ID
`model_family`	string	Model family (e.g., Llama-3.2, Qwen3)
`model_size`	string	Parameter count (e.g., 3B, 7B, 14B)
`quant`	string	Quantization method
`tier`	string	Evaluation tier (tier1 / tier2)
`task`	string	Benchmark task name
`n_shot`	float	Number of few-shot examples
`metric`	string	Metric used (e.g., acc, acc_norm)
`accuracy`	float	Task accuracy score
`stderr`	float	Standard error of accuracy
`samples_original`	float	Original number of samples
`samples_effective`	float	Effective samples after filtering

`model_inventory`

Column	Type	Description
`model`	string	Full HuggingFace model ID
`model_family`	string	Model family
`model_size`	string	Parameter count
`quant`	string	Quantization method
`has_accuracy`	bool	Whether accuracy benchmarks exist
`has_throughput`	bool	Whether throughput benchmarks exist

License

MIT