birgermoell/baby_9b_dense-iter76800-megatron

baby9bdense — Megatron checkpoint (iter 76800)

A Megatron-LM `torch_dist` training checkpoint of baby_9b_dense, the OpenEuroLLM ~9B dense (Qwen3-style) base model. This is a mid-pretraining snapshot exported as a transport/backup artifact and as the starting point for long-context extension experiments (4K → 16K → 64K → 128K).

⚠️ This is not a finished model and not a HuggingFace-format model. It is a raw Megatron distributed-checkpoint (model + distributed-optimizer state) for resuming or finetuning inside Megatron-LM. To use it as a normal HF model you must convert it first.

TL;DR for agents

artifact_kind: megatron_lm_distributed_checkpoint
checkpoint_format: torch_dist        # parallelism-agnostic: load at ANY tensor/pipeline parallel size
iteration: 76800
tokens_seen: 644245094400            # ~644B tokens (~6.4% of the 10T pretraining target)
contains: [model_weights, distributed_optimizer_state]
num_files: 2051                      # 2049 *.distcp shards + metadata.json + modelopt_run_config.yaml
total_size_gib: 119
is_final: false
intended_use: long_context_extension_continued_pretraining
load_with: Megatron-LM (pretrain_gpt.py), --ckpt-format torch_dist
tokenizer: OpenEuroLLM 256k (HuggingFaceTokenizer, vocab 262144)  # MUST match; do not substitute
compatible_longctx_data: birgermoell/oellm-longctx-tokenized-streamed-all-v2

Model architecture (exact Megatron args)

Tokenizer

OpenEuroLLM 256k tokenizer (HuggingFaceTokenizer, SentencePiece, vocab 262144, Gemma-style <bos>/<eos>/<start_of_turn>/<end_of_turn> specials). md5(tokenizer.model) = 07295117baaf7a55d5924428e59925d6. The long-context dataset `birgermoell/oellm-longctx-tokenized-streamed-all-v2` uses the same tokenizer, so it is token-ID compatible with this checkpoint.

How to use

Download

from huggingface_hub import snapshot_download
snapshot_download("birgermoell/baby_9b_dense-iter76800-megatron",
                  local_dir="checkpoints/iter_0076800")
# then create the pointer file Megatron expects:
#   echo 76800 > checkpoints/latest_checkpointed_iteration.txt

Load in Megatron-LM

Pass --load <dir-containing-iter_0076800> with --ckpt-format torch_dist and the architecture args above. Because the checkpoint is torch_dist, you may load it at any --tensor-model-parallel-size / --pipeline-model-parallel-size (it reshards on load).

• —Continued pretraining (keep optimizer/schedule): plain --load.
• —Context extension / finetune (fresh optimizer, iteration reset to 0): add --finetune, raise --seq-length + --max-position-embeddings, and increase --rotary-base (ABF) for the target context length.

Provenance

• —Project: OpenEuroLLM. Trained on Leonardo (CINECA, A100-64GB), TP=4 PP=1, GBS=2048, WSD LR schedule, 10T-token target run (production_training/baby_9b_dense).
• —This snapshot = iteration 76800.

Caveats

• —Mid-training checkpoint; quality reflects ~644B tokens, not a converged model.
• —Includes distributed-optimizer state (hence ~119 GB vs ~18 GB for weights only).
• —Not HF-format; needs Megatron→HF conversion for transformers use.