openeurollm/prelude

OpenEuroLLM Prelude 9B (WIP Research Artifact)

This repository contains intermediate training checkpoints for OpenEuroLLM Prelude, an experimental 9B dense base model currently training on Leonardo (CINECA). See the OpenEuroLLM progress tracker for live status across all OpenEuroLLM training runs. It is being trained on 10T tokens (~80% English + code + math, ~20% multilingual across the 36 initial OpenEuroLLM target languages, represented as 41 language/script codes in the data mix, since some languages are split into macrolanguage + specific-variant codes, e.g. Latvian/Standard Latvian/Latgalian, Estonian/Standard Estonian, Albanian/Tosk Albanian, Norwegian Bokmål/Nynorsk). Full dataset composition, sources, and the per-language token breakdown are documented in the training-data-collection repo.

Checkpoints are published as per-iteration branches. Each branch (e.g. iter_0124800) is a complete, self-contained HuggingFace checkpoint: config.json, sharded model-*.safetensors (5GB shards), tokenizer files, and a validation.json with example generations (see Sanity-check generations below). main itself is intentionally empty and reserved for the final checkpoint.

Note on training phase: these are checkpoints from the "stable" (constant max-LR) phase of the WSD schedule and are not cooled down. Only a checkpoint at/after iteration ~953,600 would reflect the cooldown phase; everything currently published predates that point.

Model Details

Dense, untied input/output embeddings, closely follows the Qwen3 architecture.

num tensors        : 399
dtypes             : ['BF16']
embed_tokens params: 1,074,266,112
lm_head params     : 1,074,266,112   (untied from embeddings)
transformer body   : 6,946,075,648
TOTAL params       : 9,094,607,872   (~9.095 B)
weight bytes (hdr) : 18,189,215,744  (~18.19 GB)

• —Developed by: OpenEuroLLM
• —Model type: dense, autoregressive decoder-only Transformer
• —Languages: 36 initial OpenEuroLLM target languages (41 language/script codes in the data mix, see note above); per-language token breakdown
• —License: Apache 2.0
• —Training repository: oellm-autoexp, `baby_9b_dense` experiment
• —Architecture config: `qwen3_dense_9b.yaml`

How to Get Started

Load a specific training checkpoint via the revision argument (branch name = Megatron iteration, e.g. iter_0124800):

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

repo = "openeurollm/prelude"
revision = "iter_0124800"

model = AutoModelForCausalLM.from_pretrained(
    repo, revision=revision, dtype=torch.bfloat16, device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(repo, revision=revision)

prompts = [
    "Hola, ¿qué tal?",
    "Bonjour, comment ça va ?",
    "Guten Tag, wie geht es dir?",
    "Ciao, come stai?",
    "Hei, mitä kuuluu?",
]
generate_kwargs = dict(max_new_tokens=50, do_sample=True, temperature=0.7, top_p=0.9, top_k=50, repetition_penalty=1.3, no_repeat_ngram_size=3)

for prompt in prompts:
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    outputs = model.generate(**inputs, **generate_kwargs)
    print(tokenizer.decode(outputs[0], skip_special_tokens=True))

See the repository's branch list for all available checkpoints.

Sanity-check generations

Every checkpoint's branch includes a validation.json with greedy-decoded completions for a fixed battery of prompts, generated at conversion time as a quick "does this checkpoint produce coherent text" spot-check:

• —4 canonical incomplete-sentence prompts, covering general knowledge ("The capital of France is..."), common sense ("The opposite of hot is..."), science ("Water boils at a temperature of..."), and creative/narrative ("Once upon a time...")
• —Each of the 4 is localized into all 36 target languages (144 generations total), using each language's actual native conventions where relevant (e.g. its own fairy-tale opening phrase for the narrative prompt) rather than literal word-for-word translation. English is simply one of the 36 rows, not a separate set

This is a lightweight smoke test, not a benchmark. See Evaluation for actual downstream results.

Training Details

Training logs (loss curves, throughput, etc.) are tracked on Weights & Biases.

Tokens trained at a given checkpoint

Each iteration consumes exactly one global batch, so the number of tokens seen at a given branch's iteration can be computed directly from its name:

tokens(iteration) = iteration × global_batch_size × sequence_length
                   = iteration × 2048 × 4096
                   = iteration × 8,388,608

For example, iter_0124800 corresponds to 124,800 × 8,388,608 ≈ 1.05T tokens.

Evaluation

Downstream evaluation is ongoing, see this issue.

Uses

These are intermediate, non-cooled-down research checkpoints, intended for studying training dynamics (loss/capability progression, multilingual acquisition, etc.), not for production or downstream deployment. No safety alignment or instruction tuning has been applied.