qyxu1994/babylm-2026-strict-small-baseline

babylm-2026-strict-small-baseline

BabyLM Challenge 2026 — Strict-Small track (≤10M words). Part of Reweighting Child-Directed and Conversational Data for Sample-Efficient BabyLM Pretraining (strictsmallbaseline).

• —Architecture: GPT-2-style decoder-only transformer, from-scratch init (no pretrained weights): 6 layers, 512 hidden, 8 heads, 512-token context, 27.4M parameters (tied embeddings).
• —Objective: causal language modeling.
• —Tokenizer: byte-level BPE, 16k vocab, trained only on this run's ≤10M-word training mixture.
• —Training data: 10,000,000 words (official Strict-Small corpus). Word counts use whitespace tokenization matching the official BabyLM 2026 dataset cards. Manifest and datasheet in the GitHub repo.
• —Training: 10 epochs (~100M words exposure), packed 512-token sequences, effective batch 128, lr 5e-4 cosine (3% warmup), weight decay 0.1, bf16, seed 42, single A100-80GB.

Intermediate checkpoints

As required by BabyLM 2026, checkpoints at every 1M words seen (to 10M) and every 10M (to 100M) are available as revisions chck_1M … chck_100M:

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("qyxu1994/babylm-2026-strict-small-baseline", revision="chck_5M")
tokenizer = AutoTokenizer.from_pretrained("qyxu1994/babylm-2026-strict-small-baseline")

Evaluation

Evaluated with the official babylm-eval pipeline (zero-shot: BLiMP, BLiMP-supplement, EWoK, entity tracking, COMPS, reading; fine-tuning: BoolQ, MNLI, MRPC, MultiRC, QQP, RTE, WSC):

./eval_zero_shot.sh qyxu1994/babylm-2026-strict-small-baseline causal evaluation_data/full_eval

Mixture (strictsmallbaseline)

Limitations

Single pretraining seed; 27M parameters; English only; trained on ≤10M words — a research artifact for sample-efficiency study, not a production model.

Citation

Paper forthcoming (BabyLM Challenge 2026). Please also cite the BabyLM 2026 call for papers (arXiv:2602.20092).