AlexWortega/ml-intern-moe100m-ufweb-en-step66035-10B-20260517-1222

MoE-100M @ ~10.82B tokens (Ultra-FineWeb, intermediate ckpt from a 100B run)

This is an intermediate checkpoint from a 100B-token continuation pretrain of the `ml-intern-moe100m-ufweb-en-1B` model. The full run is targeted at 100B tokens (12-day wall on 4× V100 SXM2); this snapshot is at step 66 035 (≈ 10.82 B tokens consumed total counting the 1B and 10B warm-start phases).

• —Step: 66,034
• —Tokens consumed (total): 10,819,010,560
• —Architecture: 99.35M-active MoE (12L, d=512, GQA 8/2, 8+1 experts top-2, partial RoPE, QK-Norm, SwiGLU, tied 48k vocab); see model.py.
• —Source ckpt: ckpts_100b/step_66035.pt (EMA shadow extracted for inference)
• —Trained autonomously by the ml-intern Claude Code skill.

Reproducibility bundle (TASK / PLAN / RESULTS / EVAL / train+eval logs) is in this repo. The full final ckpt at 100B will land at AlexWortega/ml-intern-moe100m-ufweb-en-100B-* when training completes (~10 days away).

Caveats

• —Trained on Ultra-FineWeb-en only — does NOT speak Russian / code / multilingual.
• —100M active params on ≤ 11B tokens — competitive with gpt2-124M and pythia-160m on 4/6 zero-shot tasks (see EVAL_10B.md in this repo), trails on lambada_openai due to limited training data variety.
• —Vocab clipped at 48 000 (gpt2 tokenizer with last 2257 ids collapsed). Don't feed token ids ≥ 48 000.