Modelpublic

AbdulRahmanIqbal/babylm-baseline-10ep

sourceHugging Facemitupdated 3mo agoView on Hugging Face

0likes72downloads

Model Card

BabyLM 2026 Strict-Small Submission — Baseline (FP32, 10 epochs)

Winning configuration from a systematic 15-way ablation of common training optimizations (mixed precision, Flash Attention, curriculum learning, dynamic batching, post-training INT8 quantization) on the BabyLM Strict-Small track. The plain FP32 baseline — no optimizations — won outright at a 10-epoch budget, statistically tied with FP16+Flash Attention.

Paper: How Much Do Common Training Optimizations Cost You? A Systematic Ablation Study on the BabyLM Strict-Small Track (BabyLM Workshop, EMNLP 2026).

Model details


Architecture	GPT-2 decoder, 33.6M params (embed=384, heads=6, layers=8, context=256)
Tokenizer	GPT-2 BPE, 50,257 vocab
Training data	BabyLM-community/BabyLM-2026-Strict-Small (10M words, detoxified)
Training budget	10 epochs (100M word exposure), plain FP32, no optimizations
Seed	789 (best of 5 validated seeds)
Optimizer	AdamW, lr=3e-4, CosineAnnealingLR (etamin=3e-5), gradclip=1.0

Branches

—main — final checkpoint (step 10,500, epoch 9), used for all full-eval and GLUE results below.
—chck_1M ... chck_9M, chck_10M ... chck_100M — 19 official word-count-milestone checkpoints (same seed/run), provided for AoA and fast-eval-across-training-steps benchmarks.

Results (n=5 seeds unless noted; see paper for full detail)

Benchmark	Score
Validation perplexity	136.786 ± 0.838 (seed 789: 136.07)
BLiMP	62.44% ± 0.47
BLiMP Supplement	58.13% ± 0.81
COMPS (base)	54.11% ± 0.35
WUGs	50.16% ± 0.22
EWoK	50.89% ± 0.71
Entity Tracking	18.54% ± 0.25
GlobalPIQA (non-parallel / parallel)	46.80% ± 3.90 / 20.00% ± 2.13
GLUE average	62.57%
AoA correlation (Pearson r)	−0.031 ± 0.070

Intended use

Research artifact for the BabyLM Challenge 2026 (Strict-Small track). Trained on a 10M-word, developmentally-plausible corpus for sample-efficiency research, not intended for production use.