alonsopg/babylm-2026-semantic-cloze-strict-small

BabyLM 2026 Strict-Small Semantic Cloze Multi-Task MLM

This is the final checkpoint from the multitask_ewok_semantic_cloze_v1 experiment for BabyLM 2026 Strict-Small.

The model is a compact BERT-style masked language model trained from scratch on the BabyLM 2026 Strict-Small corpus with a small set of auxiliary training tasks. The main added objective is an MLM-head semantic cloze ranking task: each cloze example contains a single [MASK], a good single-token completion, and a bad single-token completion. The model scores the good and bad completions with the standard MLM head.

Intended Use

This checkpoint is intended for BabyLM 2026 shared-task evaluation and for reproducing the accompanying multi-task semantic cloze experiment. It is not intended as a general-purpose language model.

Training Setup

• —Track: BabyLM 2026 Strict-Small
• —Training data: BabyLM-community/BabyLM-2026-Strict-Small
• —Tokenizer: bert-base-uncased
• —Architecture: compact BERT masked language model
• —Hidden size: 256
• —Layers: 4
• —Attention heads: 4
• —Intermediate size: 1024
• —Parameters: 11,201,338
• —Maximum sequence length: 128
• —Batch size: 96
• —Maximum steps: 10,000
• —Actual stopped step: 9,500
• —Learning rate: 5e-4
• —Weight decay: 0.01
• —Warmup fraction: 0.06
• —Precision: fp32
• —Seed: 1
• —Device used for training: NVIDIA RTX A6000

Early stopping monitored validation MLM loss with patience 2, minimum delta 0.01, and start-after step 9000.

Task Mixture

The semantic cloze ranking loss is:

softplus(-(score_good - score_bad))

where score_good and score_bad are MLM log-probabilities at the masked position. No classifier head or CLS pooling is used for this objective.

The semantic cloze examples are synthetic, single-mask examples over affordance, animate agency, part-whole, physical-property, and typical-location templates. No official EWoK or BLiMP evaluation items were used to generate the auxiliary data.

Official Fast Evaluation Snapshot

These local official fast-eval scores are from the experiment records in the submission repository.

The final checkpoint is the recommended checkpoint for EWoK-focused reporting. It improves EWoK over the MLM baseline in this local comparison, but trades off against the best-validation checkpoint on Entity Tracking.

Files

• —model.safetensors: MLM-compatible model weights
• —config.json: BERT model configuration
• —tokenizer.json, tokenizer_config.json: tokenizer files
• —training_config.yaml: training configuration
• —results_summary.json: local experiment summary

Caveats

This is a single-seed research checkpoint. The result should be presented cautiously: semantic cloze produced the best EWoK score in the isolated experiment series, while weight sweeps and relational cloze follow-ups did not improve the EWoK result further. Full official leaderboard submission artifacts still need to be generated with the BabyLM evaluation pipeline.