COPA-AI/arm-gemma-e4b

1---2license: gemma3language:4  - hy5  - en6base_model: google/gemma-4-e4b7pipeline_tag: text-generation8tags:9  - arxiv:2609.0335010  - armenian11  - continued-pretraining12datasets:13  - COPA-AI/armweb14  - COPA-AI/armstem15---16 17# arm-gemma-e4b18 19**arm-gemma-e4b** is Gemma-4-E4B adapted to Armenian by continued pretraining.20To our knowledge, it is the **first open Armenian LLM released together with21its complete training corpus and recipe** — every training token is either22public ([ArmWeb](https://huggingface.co/datasets/COPA-AI/armweb),23[ArmSTEM](https://huggingface.co/datasets/COPA-AI/armstem), FineWeb-Edu,24Stack-smol) or reproducible from the released pipeline.25 26This is a **base model**: no instruction tuning, no chat template. Use it for27Armenian text continuation, likelihood scoring, or as a starting point for28Armenian SFT.29 30## Recipe31 32- **Base**: Gemma-4-E4B, stock tokenizer (best Armenian fertility among CPT33  candidates: 4.15 tokens/word; vocabulary extension was sharply harmful in34  our ablation at this budget).35- **Continued pretraining**: 10B tokens, sequence-packed at length 4096,36  LR 3×10⁻⁵ cosine. The architectural context window is unchanged from the37  base (128K, `max_position_embeddings=131072`); CPT only exercised38  positions up to 4096, and long-context behavior beyond that is inherited39  from the base and not specifically evaluated.40- **Mixture (epoch-capped)**: 69% ArmWeb, 4% ArmSTEM-HY, 2% ArmSTEM-EN,41  20% English web replay (FineWeb-Edu), 5% code (Stack-smol) — every ArmWeb42  token seen ~2× and every ArmSTEM token 7–9× (Gemma-tokenizer accounting).43  The ArmSTEM stream is a 109,885-item subset of the corpus (~30%, sampled44  at random with balanced stratification across the math and science45  pools); its manifest ships with ArmSTEM (`stats/cpt_training_subset_ids.txt`,46  104,630 items verbatim in the release, 5,255 superseded before release).47  A control run trained on the full 373K corpus (~1.3 epochs instead of48  7–9) scores within noise of the released model (paper §5), so the49  repetition is costless.50- **Details**: global batch 512 sequences (~2.1M tokens/step), 4,770 steps,51  100 warmup steps, AdamW (β₂=0.95), weight decay 0.1, bf16, seed 42.52 53The mixture is the headline finding of the accompanying paper: news-only CPT54catastrophically forgets (−21.2pp Belebele at LR 10⁻⁴); a gentler LR recovers55two-thirds; **6% verified translated STEM data reverses forgetting entirely**,56ending +2.2pp above the unadapted base while keeping the fluency gains.57 58## Results59 60Six-task Armenian likelihood suite (accuracy; harness: lm-eval):61 62| Task | Gemma-4-E4B (base) | **arm-gemma-e4b** |63|---|---|---|64| Belebele-hye | 0.619 | **0.716** |65| INCLUDE-Armenian | 0.416 | **0.456** |66| m-MMLU-hy | 0.343 | 0.337 |67| ARC-hy | 0.227 | **0.229** |68| HellaSwag-hy | 0.266 | 0.262 |69| MultiBLiMP-hye | 0.989 | **0.992** |70| **Mean** | 0.477 | **0.499** |71 72This is the highest six-task mean among all open Armenian models we73evaluated — the best prior models score 0.471 (ArmenianGPT-1.0-3B) and740.436 (HyGPT-10b), both *below* the unadapted base.75 76ArmBench-LLM generative tasks (base-model-appropriate metrics; full77results in the paper appendix):78 79| Task | base | **arm-gemma-e4b** |80|---|---|---|81| SynDARin (EM) | 0.04 | **0.92** |82| Hartak (EM) | 0.02 | **0.82** |83| DREAM (EM) | 0.48 | **0.84** |84| Belebele gen. (EM) | 0.66 | **0.90** |85| Scientific MCQA (EM) | 0.86 | **1.00**\* |86| MMLU-Pro-Hy | 0.154 | **0.251** |87| Topic (14-class) | 0.004 | **0.482** |88 89\*Audited: zero shared 8-grams between the benchmark items and the ArmSTEM90training corpus (paper, Appendix).91 92**Honest negatives**: POS tagging regresses under CPT (0.18→0.01); exam93mathematics is flat for the released model (1.75 points), and the94full-corpus control run reaching 2.75 suggests data diversity rather than95difficulty is the binding factor; instruction-dependent96ArmBench tasks (judged generation, BLEU QA) are low for all base-style models97including this one — they measure formatting, and we defer them to an98instruction-tuned variant.99 100## Training data & contamination101 102Training data was decontaminated by 13-gram overlap against ten Armenian103evaluation sets and all ArmBench items, on both the English and Armenian104side for translated data. The two remaining reported benchmarks, m-MMLU-hy105and ARC-hy, were scanned post hoc: zero of 4.31M ArmWeb training106documents and 4 of 372,907 ArmSTEM pairs share any 13-gram with their107items. Existing public Armenian corpora overlap these benchmarks at1087.9–17.4% (see the [ArmWeb card](https://huggingface.co/datasets/COPA-AI/armweb)).109 110## Limitations111 112Inherits Gemma-4's biases and terms of use. News-domain-concentrated Armenian113exposure. Machine-translated STEM data verified for answer preservation and114language identity, not stylistic fluency (though a two-annotator115native-speaker audit rated 299 of 300 sampled problems valid, Cohen's116κ = 1.0). No safety tuning.117 118## Citation119 120```bibtex121@article{arakelyan2026armweb,122  title  = {From Zero to Hero: An Open LLM Ecosystem for Armenian},123  author = {Arakelyan, Erik and Avetisyan, Khatun and Davtyan, Meri and Grigoryan, Heghine and Khachatryan, Nane and Shahsuvaryan, Hayk and Sergoyan, Henrik and Martirosyan, Vahan},124  year   = {2026},125  journal = {arXiv preprint arXiv:2609.03350}126}127```128