Modelpublic

The-CoLab/llama3-7b-en-ru-v2

sourceHugging Facellama3updated 2mo agoView on Hugging Face

0likes9downloads

Model Card

llama3-7b-en-ru-v2

LLaMA-3 7B pretrained on English-Russian bilingual data, 134k steps (injection fix v2).

Validation Results

Final checkpoint: step 133,600

Validation set	Cross-entropy loss (nats)	Perplexity
English (en)	2.2403	9.40
RU (`ru`)	0.7749	2.17

Note: Rerun with corrected injection config (Buckwalter transliteration fix + injection count fix). Training proceeded in two LSF jobs: the first (step 38,000–72,501) was restarted due to a node issue; the second continuation (step ~74,010–133,600) ran cleanly to completion. The final validation losses above are from step 133,600 of the continuation job. An earlier intermediate checkpoint at step 70,808 had EN=1.273 / RU=0.941 (reported in a prior version of this card).

Evaluation Results

EEE-format evaluation results are stored under eval_results/eee/ in this repository. Tasks: Global MMLU (EN/RU), PIQA, ECLeKTic, Fictive Entity (2-rate mix).

Training Curves

Per-step training and validation loss curves are stored in training_curves/:

File	Description
`train_loss.csv`	Training loss every 10 steps (`step,train_loss`) — covers the continuation phase (~steps 42,500–133,600)
`val_loss.csv`	Validation loss per language (`step,lang,val_loss`) — 90 rows, steps 74,816–133,600 (continuation phase)

Citation

Part of the The-CoLab multilingual-transfer collection.