Datasetpublic

ZhuofengLi/megatron-configs

Training model configurations This directory records model architecture configurations used by training examples or imported from reference experiments. These files preserve model structure and provenance; they do not automatically configure or launch a Megatron Bridge training job. Configurations The H100 entries below are restricted to runs pinned in the RSI-Index/ale-rsi-pre-training workspace. Nominal Compute Model Parameters Actual Tokens Training GPUs… See the full description on the dataset page: https://huggingface.co/datasets/ZhuofengLi/megatron-configs.

sourceHugging Faceupdated 2mo agoView on Hugging Face

0likes323downloads

Dataset Card

Training model configurations

This directory records model architecture configurations used by training examples or imported from reference experiments. These files preserve model structure and provenance; they do not automatically configure or launch a Megatron Bridge training job.

Configurations

The H100 entries below are restricted to runs pinned in the `RSI-Index/ale-rsi-pre-training` workspace.

Nominal Compute	Model	Parameters	Actual Tokens	Training GPUs	Pure Training Time	End-to-End Wall Time	Model Config	W&B Run
9e18	d1152-L12	550.3M	2.904B	8×H100	2.10 h	4.58 h	GBS16 / MBS2 / GA1 / offload=1	5h5yihr2
1.8e19	d1408-L15	837.0M	3.613B	8×H100	3.31 h	7.26 h	GBS16 / MBS2 / GA1	952nr4js
3e19	d1536-L16	998.0M	4.983B	8×H100	3.82 h	6.56 h	GBS32 / MBS4 / GA1	3hz3kt5j
9e19	d1792-L18	1.385B	10.560B	8×H100	9.23 h	12.75 h	GBS64 / MBS2 / GA4	lonea36p
1.8e20	d2048-L21	1.935B	14.805B	8×H100	~16.44 h projected	13.40 h observed at step 35,077/56,477; ~21.9 h projected full	GBS64 / MBS2 / GA4	76xmgi2r
3e20	d2048-L21	1.935B	24.675B	16×H100	21.56 h	24.49 h	GBS128 / MBS2 / GA4	yc0dfezu

Pure training time is computed from the median stable iteration time after step 100 multiplied by the configured training steps; it excludes LM Evaluation Harness. End-to-end time includes Harness evaluations, checkpointing, initialization, and final sync, but excludes scheduler queue time. Run 76xmgi2r did not finish, so its full-run values are projections from the observed iterations. Run yc0dfezu lost W&B connectivity at step 26,050, while the same training job completed all 47,064 steps in 24.49 hours according to the scheduler and final success marker.

All pinned H100 layouts use TP=PP=CP=1, no activation recomputation, and fused gradient accumulation. The 550M pinned run has activation_offload_fraction=1 in its W&B config despite nooffload appearing in its display name; all other pinned runs use zero activation offload. The Marin source configuration remains recorded separately in each YAML.

The Marin source YAML preserves the inherited model.use_qk_norm: false value emitted by LlamaConfig. These runs instantiate Qwen3Config, whose attention_config() overrides that inherited field and always installs learned per-head Q/K RMSNorm. Therefore each file records resolved.normalization: rmsnorm and resolved.effective_use_qk_norm: true to distinguish the effective model architecture from the serialized source field.

When adapting a reference configuration to Megatron Bridge, keep the source model fields unchanged unless the experiment explicitly studies architecture changes. Record training schedule, optimizer, data, and distributed parallelism separately because those settings are framework- and hardware-dependent.