Modelpublic

1ghub/babylm-2026-mtl-bigckpt

sourceHugging Facecc-by-nc-4.0updated 16d agoView on Hugging Face

0likes231downloads

Model Card

babylm-2026-mtl-bigckpt

A 213M-parameter GPT-2-style causal language model submitted to the multilingual track of the BabyLM 2026 Challenge (English, Dutch, Chinese), trained from scratch within the 100M English-equivalent-word budget.

This is the overall submission. It obtained a leaderboard-reported Multilingual Average of 50.68.

Model details


Architecture	GPT-2-style decoder with multi-token prediction (MTP)
Parameters	212,790,272 (213M)
Layers / heads / hidden	14 / 16 / 1024
Context length	512
Tokenizer	byte-level BPE, 32,000 vocab
Training objective	next-token + MTP auxiliary heads ($k{=}4$, ramped $1\to4$ over epochs; only the first head is used at inference)
Optimizer	AdamW, peak LR $4.5\times10^{-4}$, 10 epochs
Track	BabyLM 2026 multilingual
Languages	English, Dutch, Chinese

Training data

Trained within the fixed multilingual budget on a combination of:

—the official BabyBabelLM corpora for English, Dutch, and Chinese;
—parallel-corpus text from Tatoeba and OPUS-100;
—task-family-targeted synthetic text generated offline by an approved teacher model (Qwen3.5-4B).

Published training data: **`1ghub/babylm-2026-mtl-data`**

Note on budget. The track measures the budget in bytes: the 100M-word tier is 543.00 MB, and a corpus qualifies while its UTF-8 size divided by the language premium stays at or below that. We use the coefficients published in the organizers' table: English 1.0, Dutch 1.0516, and simplified Chinese 0.93597, the script BabyBabelLM uses. (The 0.9894 listed in the call for papers is the coefficient that table gives for traditional Chinese.) Under this rule the training corpus is 527.7 MB of English-equivalent bytes, 97.2% of the cap.

Results

Development-set scores (percentages):

Metric	Score
Official Multilingual Average	50.68
Zero-shot (16-task mean, ZS16)	57.82
Fine-tuning (22-task mean, FT22)	41.25
ZS — English / Dutch / Chinese	59.89 / 59.77 / 53.41
FT — English / Dutch / Chinese	43.86 / 39.37 / 40.61

Intended use and limitations

Research model for data-efficient multilingual language modeling under the BabyLM setting. It is a small model trained on a developmentally plausible amount of text and is not intended for production use. Scores are on development sets; evaluation reuse across selection stages introduces selection bias.

Teacher model

Synthetic training text was generated (not distilled) by Qwen3.5-4B, an approved teacher for the challenge. Generation prompts used task-level descriptions.

Citation

Companion paper: Language Allocation and Targeted Data Design for a Multilingual BabyLM (BabyLM 2026 Workshop, EMNLP 2026).