AlexWortega/ml-intern-v4-100m-tinystories-demo

ml-intern MoE-100M demo

This Space hosts demos for the `ml-intern` MoE-100M family pretrained autonomously on openbmb/Ultra-FineWeb by the ml-intern Claude Code skill.

Latest model — ml-intern-moe100m-ufweb-en-step66035-10B-20260517-1222

Intermediate checkpoint from an in-flight 100B-token continuation pretrain (warm-started from the 10B baseline). Snapshot at step 66 035 ≈ 10.82B tokens consumed total.

• —Architecture: 12-layer MoE (d=512, GQA 8/2, 8+1 experts top-2, layer-0 dense, partial RoPE, QK-Norm, SwiGLU, tied 48k vocab) — 99.35M active params, 229M total
• —Final run will land at AlexWortega/ml-intern-moe100m-ufweb-en-100B-* (~10 days out)
• —Trained on 4× Tesla V100-SXM2 32GB, fp16 AMP, Muon + AdamW, WSD schedule with EMA
• —Zero-shot beats gpt2-124M / pythia-160m on 4 of 6 tasks (BoolQ, HellaSwag, PIQA acc_norm, ARC-Easy) — see EVAL.md in the model repo

→ Open model card

Earlier model in the same family — ml-intern-moe100m-ufweb-en-1B-20260515-0823

The 1B-token shakedown that proved the pipeline end-to-end (WSD + EMA + ckpt + SIGTERM-resume + eval + HF push). Same architecture, same code, smaller token budget.

→ Open model card

Browser inference

The static HTML+JS shell below currently runs TinyStories-33M ONNX q4 via @huggingface/transformers + WebGPU. The MoE-100M family is fp16 PyTorch — porting it to client-side WebGPU requires an ONNX export pipeline + q4 / q4f16 quantization of the MoE dispatch (router top-k + grouped FFN), which is non-trivial for a custom MoE block and is on the to-do list. Until that's done, the WebGPU demo here serves TinyStories-33M; for the MoE-100M, use the model cards above directly with PyTorch.

Files

Browser support (TinyStories-33M demo only)

• —WebGPU: Chrome ≥113, Edge ≥113, Arc, Safari Technology Preview ≥17.4
• —Fallback: WASM (slower; used automatically when navigator.gpu is missing)