Modelpublic

exusiaiw/emnlp-babylm-2026

sourceHugging Facemitupdated 3mo agoView on Hugging Face

0likes22downloads

Model Card

EMNLP BabyLM 2026 — Multilingual OLMo2

Submission for the BabyLM 2026 Challenge Multilingual Track (EMNLP 2026 Workshop).

Model Description

A trilingual causal language model (English / Dutch / Chinese) pretrained from scratch on developmentally plausible text data, using the OLMo2 architecture.

—Architecture: Olmo2ForCausalLM
—Parameters: ~15M (512h / 8L / 8H / 1376ff)
—Tokenizer: BPE, vocab=32000
—Sequence length: 512 tokens
—Languages: English, Dutch, Chinese (equal data mix)
—Data: BabyLM 2026 multilingual corpus (100M tokens total)

Results

Multilingual Track leaderboard (as of submission):

Model	Multilingual Avg	EN Avg	NL Avg	ZH Avg
This model	~45-47	-	-	-
Best baseline (gpt2-ennldzho_equal)	38.86	40.04	40.02	36.51

Selected task scores:

—MultiBLiMP: EN 84.7%, NL 92.4%
—ZhoBLiMP: 78.5%
—SIB-200: EN 81.0%, NL 77.5%, ZH 77.0%
—MNLI: EN 57.8%, NL 57.3%, ZH 57.1%

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("exusiaiw/emnlp-babylm-2026")
tokenizer = AutoTokenizer.from_pretrained("exusiaiw/emnlp-babylm-2026")

Citation

BabyLM 2026 Challenge — EMNLP 2026 Workshop, Budapest