Datasetpublic

Glow-AI/pco64_tabular_data

pco64 Throughput of distributed LLM training as a function of the parallelism configuration, on 64 GPUs across 16 hosts. Tabular benchmark for the bolt problem pco64: the 780 rows are the full candidate set. Objective throughput_mean, to maximise. NaN where the run OOMed, since no throughput is observed at all -- a hidden (crash) constraint, not a bad value. Constraint ran_successfully. 560 of 780 configurations run; the rest exhaust GPU memory. Feasibility is only learnable by… See the full description on the dataset page: https://huggingface.co/datasets/Glow-AI/pco64_tabular_data.

sourceHugging Facemitupdated 20h agoView on Hugging Face

0likes26downloads

Dataset Card

pco64

Throughput of distributed LLM training as a function of the parallelism configuration, on 64 GPUs across 16 hosts. Tabular benchmark for the bolt problem pco64: the 780 rows are the full candidate set.

—Objective throughput_mean, to maximise. NaN where the run OOMed, since no throughput is observed at all -- a hidden (crash) constraint, not a bad value.
—Constraint ran_successfully. 560 of 780 configurations run; the rest exhaust GPU memory. Feasibility is only learnable by attempting the run, so it cannot be decoupled from the objective.
—Auxiliary peak_mem (GB, inf when infeasible), peak_mem_normalised (the run's own feasibility margin, < 0 iff feasible, a fixed pad value when infeasible) and overall_time (wall clock in seconds, NaN when infeasible).

peak_mem does not define the constraint: feasible runs here reach 79.3 GB, so no memory threshold separates the classes. peak_mem_normalised does, by construction.

Instance context

Constant for every row; these describe the instance, not the search space.

field	value
`num_layers`	64
`num_attention_heads`	64
`hidden_size`	5120
`ffn_hidden_size`	25600
`batch_size`	512
`seq_length`	8192
`log2_num_gpus`	6
`log2_num_hosts`	4
`log2_grad_accum_steps`	0

Columns

The table is the raw sweep export, unchanged. Every knob halves or doubles, so the sizes arrive as log2_* and their levels are evenly spaced. zero_stage is the raw stage (0/2/3); bolt searches it as an ordinal 0/1/2.

column	role	meaning
`log2_dp_size`	input	log2 of data-parallel size
`log2_tp_size`	input	log2 of tensor-parallel size
`log2_pp_size`	input	log2 of pipeline-parallel size
`log2_cp_size`	input	log2 of context-parallel size
`log2_sp_size`	input	log2 of sequence-parallel size
`log2_dp_bucket_size_mb`	input	log2 of DDP gradient bucket size in MB
`zero_stage`	input	ZeRO stage, one of [0, 2, 3]
`log2_num_model_chunks`	input	log2 of number of model chunks
`log2_grad_accum_steps`	context	log2 of gradient accumulation steps
`ran_successfully`	output	ground-truth constraint label
`throughput_mean`	output	objective, `NaN` when infeasible
`throughput_std`	output	spread over the timed steps, `NaN` when infeasible
`peak_mem_normalised`	output	feasibility margin, `< 0` iff feasible
`overall_time`	output	wall clock in seconds, `NaN` when infeasible
`all_time_steps`	output	per-step times, as a string list
`peak_mem`	output	peak GPU memory in GB, `inf` when infeasible
`peak_mem_per_gpu`	output	per-GPU peak memory, as a string list

Best observed

throughput_mean = 0.27068 at log2_dp_size=4, log2_tp_size=2, log2_pp_size=0, log2_cp_size=0, log2_sp_size=2, log2_dp_bucket_size_mb=6, zero_stage=2, log2_num_model_chunks=0 (peak memory 74.2 GB).

Provenance

data_64depth-64gpus.csv, converted to parquet unchanged.