leslie721007/babylm-strict-small-scale1p75-chck82

BabyLM Strict-Small scale-1.75 adapter checkpoint at 82M words

This repository contains the protected BabyLM Strict-Small endpoint selected from a legal 100M-word training trajectory: a DeBERTa-v2 masked language model with zero-output residual bottleneck adapters inserted after each encoder layer and trained with adapter scale 1.75. The checkpoint is the chck_82M function with 35,463,008 parameters and exactly 82,012,495 counted training-word exposure. It was trained from random initialization with a tokenizer fitted only on the same 10M-word submitted corpus.

The model must be loaded with trust_remote_code=True. Loading with vanilla AutoModelForMaskedLM(..., trust_remote_code=False) instantiates native DebertaV2ForMaskedLM and ignores the 48 adapter tensors, yielding a different 34,467,424-parameter function. That native fallback is not this model.

from transformers import AutoModelForMaskedLM, AutoTokenizer
repo_id = "leslie721007/babylm-strict-small-scale1p75-chck82"
revision = "main"
tok = AutoTokenizer.from_pretrained(repo_id, revision=revision)
model = AutoModelForMaskedLM.from_pretrained(repo_id, revision=revision, trust_remote_code=True)
model.eval()

Architecture and training coordinate

• —Base architecture: DeBERTa-v2 MLM, 8 layers, hidden size 480, 8 attention heads, FFN size 1920, relative attention enabled.
• —Residual pathway: per-layer bottleneck adapter with layer norm, down projection 480→128, GELU, up projection 128→480, post-layer addition scaled by 1.75.
• —Parameters: 35,463,008.
• —Tokenizer: 16,384-entry ByteLevel BPE fitted only on the submitted 10M-word training pool; tokenizer.json SHA256 in the local run record is 91b775514b9f4e2d1f37c28445ab98181007e16d14f763955168547e293ee8f9.
• —Corpus: BabyLM Strict-Small-compatible 10M-word compact-view reinvestment pool. The pool combines official BabyLM-source material, counted Qwen-generated semantic second views, and reinvested source diversity under the same 10M budget.
• —Exposure: 82,012,495 counted words, within the BabyLM Strict-Small 100M-word / 10-epoch allowance.
• —Optimizer: AdamW using the session's fixed 100M-horizon schedule and whole-word masking recipe.

Official-compatible evaluation

The checkpoint was independently verified from the reproduced artifact and from a from-corpus bit-identical reproduction. The complete official-compatible 2026 Strict-Small evaluation gives:

The public leaderboard snapshot used during packaging displayed a top Strict-Small Overall score of 41.80; this endpoint's verified Overall is 41.942481167385985. The score carrier submission/all_full_preds_and_fast_scores_mlm.json contains the full and fast-evaluation prediction blocks used for leaderboard submission validation; its SHA256 is dcad3d8cf285a69d31910a32c62c80689022a3ff459a0956401ab7f4b3542237.

Provenance files in this repository

• —provenance/score_summary.json: score vector, corpus/tokenizer/model hashes, and endpoint identities.
• —provenance/step131_chck82_independent_verification.json: A02 independent score/provenance/loadability verification.
• —provenance/step167_from_corpus_reproduction_compare.json: A01 from-corpus bit-identical reproduction check.
• —provenance/step171_fast_submission_verification.json: fast-inclusive prediction-carrier validation.
• —provenance/step172_fast_eval_input_provenance.json: fast-evaluation input provenance.
• —submission/all_full_preds_and_fast_scores_mlm.json: official-compatible prediction file with full and fast results.

Use and scope

This is a research checkpoint for BabyLM Strict-Small evaluation and reproducible analysis. It should not be treated as a general-purpose language model. Users must comply with BabyLM rules and the terms of the underlying data sources. The custom adapter modeling code is included so that the published checkpoint loads to exactly the scored function.