Modelpublic

openeurollm/datamix-9b-Dolci-Translated-A-75EN

sourceHugging Faceapache-2.0updated 4mo agoView on Hugging Face

0likes24downloads

Model Card

datamix-9b Dolci-Translated A-75EN

Cross-architecture validation of the Translate, Replay, Mix recipe applied to openeurollm/datamix-9b-80-20, a Llama-architecture, multilingually-pretrained $9$B base with a Gemma tokenizer. Same $75/25$ English replay + Dolci-Translated EU mixture as openeurollm/OLMo-3-7B-Dolci-Translated-A-75EN, but trained with a different framework and at a different context length because of the base's architecture and pretraining limits.

Recipe


Base checkpoint	`openeurollm/datamix-9b-80-20` ($9$B Llama, Gemma tokenizer, vocab 262k, $2048$ context)
English half (Dolci replay)	`allenai/Dolci-Instruct-SFT`, 75% of the mixture
EU half (Dolci-Translated)	`openeurollm/Dolci-Instruct-SFT-translated`, 25%, 7 EU languages translated with `gemma-3-27b-it`
EU languages	cs, de, es, fi, fr, it, sv
Total samples	2.87M (same mixture as OLMo A-75EN)
Final step	41000
Chat template	`simple_chat` (no built-in template on the base; use `--chat_template_name simple_chat` or apply manually)

Training configuration

Departs from the OLMo runs in five places (see Section 3.3 of the paper):

—Llama-architecture $9$B base (vs OLMo)
—Maximum context length $2048$ (vs $32$k)
—Micro-batches accumulated to effective batch of $128$ (vs ${\sim}1$M tokens)
—Plain DDP (vs DeepSpeed ZeRO 2)
—open-instruct/finetune.py entry point (vs OLMo-core)

Peak LR $8\times10^{-5}$ matches the OLMo runs.

Evaluation

Per-language Bradley-Terry Elo at matched $1024/1024$ input/output truncation, 500 battles/language, against openeurollm/OLMo-3-7B-Dolci-Translated-A-75EN re-evaluated under the same truncation (paper Table 4, Figure 6):

Model	en	cs	de	es	fi	fr	it	sv
This repo (`datamix-9b-Dolci-Translated-A-75EN`)	$879 \pm 16$	$\mathbf{833 \pm 15}$	$\mathbf{792 \pm 19}$	$764 \pm 20$	$\mathbf{815 \pm 35}$	$790 \pm 18$	$780 \pm 18$	$\mathbf{820 \pm 32}$
OLMo-3-7B A-75EN	$\mathbf{970 \pm 14}$	$733 \pm 18$	$742 \pm 20$	$\mathbf{820 \pm 17}$	$752 \pm 38$	$\mathbf{804 \pm 16}$	$\mathbf{821 \pm 15}$	$786 \pm 32$

This checkpoint leads on cs ($+100$), de ($+50$), fi ($+63$), sv ($+34$); OLMo-3-7B A-75EN leads on en ($+91$), es ($+56$), it ($+41$), fr ($+14$). The crossover is consistent with a multilingually-pretrained base partially substituting for the multilingual SFT pool on lower-resource EU languages.

How to load

python

from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained("openeurollm/datamix-9b-Dolci-Translated-A-75EN")
model = AutoModelForCausalLM.from_pretrained("openeurollm/datamix-9b-Dolci-Translated-A-75EN", torch_dtype="bfloat16")
# No built-in chat template; use 'simple_chat' from open-instruct or apply your own.

Citation

Please cite the paper if you use this checkpoint.