DruidTheGetafix/apiary-7B-A1B

apiary-7B-A1B

A 64-expert Mixture-of-Experts language model pretrained from scratch on 9.84B tokens of FineWeb-Edu in 5.24 hours on one 8xB200 node (41.9 B200 GPU-hours). Architecture is exactly Qwen3MoeForCausalLM (16 layers, d=2048, 16/8 heads, 64 experts top-8, expert size 1024, vocab 49152): 6.85B total / 1.21B active parameters. Code, training log and configs: https://github.com/Gaurav-Shah05/apiary

This is a research artifact from a fixed compute budget: it is a coherent base LM in the GPT-2-XL / Pythia-1B class, not an instruction model and not competitive with models trained on trillions of tokens.

Training

• —Data: first 20 parquet files of HuggingFaceFW/fineweb-edu sample/100BT, tokenized with the SmolLM2 tokenizer (EOS-separated, no document masking), consumed sequentially — see data.py in the code repo.
• —Optimizer: AdamW lr 3e-4 (betas 0.9/0.95, wd 0.1), 1.05M-token batches, 500 warmup steps, constant then linear decay over the last 20% of the wall-clock budget; grad clip 1.0; load-balance loss 0.01, router z-loss 1e-3.
• —Systems: PyTorch 2.13, FSDP2 + torch.compile + grouped_mm experts, bf16 with fp32 master weights; median 613k tokens/s (28.2% MFU) on 8xB200; 9382 optimizer steps.
• —Final train CE 2.497; held-out FineWeb-Edu CE 2.462 (ppl 11.7) on 262k unseen tokens.

[image]

Evaluation (lm-eval-harness, 0-shot)

(acc_norm where the task defines it.) Logits parity vs the training code: argmax agreement 90.6%, mean |diff| 0.077.

Provenance

Weights sha256 (first 16 hex): {"model-00001-of-00004.safetensors": "f9266617b050d0fa", "model-00002-of-00004.safetensors": "cf862b24644ea799", "model-00003-of-00004.safetensors": "b09df28b263a22f6", "model-00004-of-00004.safetensors": "7ea4c3d916870908"}. Full per-10-step training log: metrics.jsonl.