Modelpublic

leslie721007/babylm-strict-small-scale1p75-chck84

sourceHugging Faceotherupdated 25d agoView on Hugging Face

1likes43downloads

Model Card

BabyLM Strict-Small scale-1.75 adapter checkpoint at 84M words

Repository target: leslie721007/babylm-strict-small-scale1p75-chck84 Revision hint at construction time: main

This repository contains the exact chck_84M function from session S0829A02: a legal BabyLM Strict-Small DeBERTa-v2 masked language model with function-preserving residual bottleneck adapters trained at fixed adapter scale 1.75. It is a same-trajectory endpoint from the reproducible scale1.75 seed43022 run, selected after the dense late-grid evaluation found a narrow broad-competence peak around 84M counted words.

Loading requirement

Use trusted code:

python

from transformers import AutoModelForMaskedLM, AutoTokenizer
repo_id = "leslie721007/babylm-strict-small-scale1p75-chck84"
revision = "main"
tok = AutoTokenizer.from_pretrained(repo_id, revision=revision)
model = AutoModelForMaskedLM.from_pretrained(repo_id, revision=revision, trust_remote_code=True)
model.eval()

Without trust_remote_code=True, Transformers can instantiate native DebertaV2ForMaskedLM and ignore the 48 adapter tensors. That native fallback has fewer parameters and is not this scored function.

Exact endpoint identity

—Model class: AdapterDebertaV2ForMaskedLM
—Parameters with trusted code: 35,463,008
—Adapter bottleneck: 128
—Adapter scale: 1.75
—Target checkpoint label: chck_84M
—Actual counted-word exposure: 84,028,405 words (8.4028405 epochs against a 10M-word pool)
—model.safetensors SHA256: 2217917c687faf4de26ef6f381be3048d0bd66b2025c06382683c24d78e8d8c9
—checkpoint-carried tokenizer.json SHA256: a9cbb830495cb92bbb2996adc256207746282ee67f4f8d40a4f646a634ec139a
—raw Step35 legal-tokenizer training artifact SHA256: 91b775514b9f4e2d1f37c28445ab98181007e16d14f763955168547e293ee8f9
—Legal 10M pool SHA256: 215944978157394dbecf2039f2c1e1806bfcbb9701421920f78605eb58975a23
—100M stream SHA256: 3dd19f09deeca44d6b07340e70cd15baa4b5c7bffe0bd462ff37536e470e7691

Current local official-compatible scores

The current session has selected full cheap-task predictions and a full SuperGLUE run for this endpoint:

column	score
Overall projection with AoA=0	42.018912974218
BLiMP	68.250000
Supplement	63.480000
EWoK	50.070000
Entity	28.580000
COMPS	52.210000
GlobalPIQA	38.120000
Reading	8.155000
SuperGLUE	69.305216767963
AoA used only for this arithmetic projection	0.000000

The arithmetic Overall projection 42.018912974218 uses AoA=0.0, matching the session's established AoA behavior for this model family. It is an endpoint-comparison coordinate, not a complete leaderboard submission package. No new leaderboard submission is included in or triggered by this repository.

Compared with the protected submitted chck_82M endpoint, this endpoint has +0.076431806832 projected Overall under the same AoA=0 arithmetic. Compared with coherent86 alpha0.75, it is -0.102111735749 lower in projected Overall but is a cleaner ordinary-training checkpoint rather than a private-scale endpoint interpolation.

Scientific status

chck_84M is an endpoint branch inside one seed/mask trajectory. Session analyses show it is a real but narrow late competence-allocation peak: it improves several cheap-task families over chck_82M, while many item gains are transient by 86M and the 100M endpoint declines despite continuing MLM-loss improvement. The next scientific decision is therefore cross-trajectory robustness, especially the seed43122 common-grid scoring and paired-session directional evidence. This repository carries the exact model function for inspection and reuse; it does not establish that the late peak is a general learning law.