leslie721007/Qiushi-BabyLM-35M-Strict-Small-v3

Qiushi-BabyLM-35M Strict-Small v3

Qiushi-BabyLM-35M-Strict-Small-v3 is a 35.46M-parameter English masked language model developed autonomously by Qiushi Engine for sample-efficient pretraining under the BabyLM Strict-Small setting.

The model combines compact-view reinvestment with zero-output residual adapters. Compact semantic views preserve useful parallel experience while returning word budget to diverse natural text. The adapters add a learnable low-rank residual pathway to each DeBERTa-v2 encoder layer while preserving the base function at initialization.

Model

Training

The tokenizer was trained on the same 10,000,000-word corpus used for model pretraining. The model was initialized from random weights and the released checkpoint was selected at exactly 82,012,495 counted word exposures.

Training used AdamW, a peak learning rate of 0.001, 6% linear warmup followed by cosine decay, an average batch size of 65,536 tokens, and seed 43022. The corpus combines BabyLM-source material, counted compact semantic views generated with Qwen3.5-9B, and reinvested natural-text diversity. The student received text only; no teacher weights, hidden states, logits, or tokenizer were transferred.

Evaluation

Complete BabyLM 2026 Strict-Small official-compatible evaluation:

The checkpoint was independently reproduced from the training corpus with an identical model SHA-256 and identical logits. The full prediction carrier also includes all required fast-checkpoint evaluations.

Usage

The adapter implementation is part of the model, so trusted repository code must be enabled:

from transformers import AutoModelForMaskedLM, AutoTokenizer

repo_id = "leslie721007/Qiushi-BabyLM-35M-Strict-Small-v3"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForMaskedLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
)

text = f"The child {tokenizer.mask_token} the book."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)

This is a research masked language model rather than an instruction-tuned chat model.

Integrity

SHA256SUMS records the exact public artifact identities. Credentials, private system code, local paths, internal traces, and operational logs are excluded. See METHOD.md for the public method summary.