leslie721007/Qiushi-BabyLM-36M-Strict-Small-v4

Qiushi-BabyLM-36M-Strict-Small-v4

Qiushi-BabyLM-36M-Strict-Small-v4 is a 36.46M-parameter masked language model developed by Qiushi Engine for BabyLM 2026 Strict-Small. Qiushi Engine drove the research loop from hypothesis formation and controlled ablations through training, evaluation, and reproducible release preparation.

Result

The complete local official-compatible evaluation is:

The Fast checkpoint package contains all 133 expected prediction records, and AoA was measured over 19 checkpoints with 8,005 contexts per checkpoint. This is a local official-compatible result; leaderboard identity is established only after separate server-side submission and scoring.

Evaluation and checkpoints

• —The complete full-plus-Fast prediction bundle is published under submission/.
• —The 19 official Fast checkpoints are published as Hugging Face revisions: chck_1M through chck_10M, followed by chck_20M through chck_100M in 10M increments.
• —See CHECKPOINTS.md, TRAINING.md, and DATA.md for the public provenance and reproduction description.

Research contribution

The model combines two evidence-backed ideas found through Qiushi Engine's research:

1. 1.Compact-view reinvestment. Meaning-preserving compact views reduce redundant wording, and the recovered fixed word budget is reinvested in diverse natural-language experience. The full training pool remains within 10M words.
2. 2.Dual-path residual consolidation. An existing residual pathway preserves broad competence while a zero-output residual pathway is attached and trained during a short coherent continuation. Its calibrated contribution balances adaptation against retention without replacing the original pathway.

The tokenizer has 16,384 entries and was trained only on the same counted 10M-word training pool. The released endpoint represents 86,005,295 counted word exposures. Compact views were generated from training-corpus sentences with Qwen3.5-9B and were counted inside the data budget.

Architecture

• —DeBERTa-v2 encoder: 8 layers, hidden size 480, 8 attention heads
• —Two 128-dimensional bottleneck residual adapters per encoder layer
• —36,458,592 parameters
• —Masked language modeling with whole-word masking
• —Maximum training sequence length: 256 tokens

Usage

from transformers import AutoModelForMaskedLM, AutoTokenizer

repo = "leslie721007/Qiushi-BabyLM-36M-Strict-Small-v4"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForMaskedLM.from_pretrained(repo, trust_remote_code=True)
model.eval()

trust_remote_code=True is required because the residual architecture extends the native Transformers DeBERTa-v2 implementation.

Scope and limitations

This is an English masked language model for representation learning, scoring, and fill-mask experiments. It is not an instruction-tuned conversational model. The release documents one high-performing endpoint; broader claims about the general mechanism require additional seeds and model families.