ypwhere/attnres4babylm2026-10m-l16-dynamic-s1338

Block Attention Residuals · 10M · 16 layers · seed 1338

This model accompanies What Does Input-Dependent Depth Routing Buy in Data-Limited Language Modeling? for BabyLM 2026. It is a causal language model trained on the official BabyLM 10M-word training split, intended for research on language modeling with limited data and depth routing.

Each target sublayer computes input-dependent softmax weights over depth-wise source representations. Completed residual blocks provide additional sources.

All 21 models

Code and reproduction instructions

Architecture

The shared backbone uses RMSNorm, SwiGLU, rotary positional embeddings, gated attention, and tied input/output embeddings.

Training

The training track denotes the size of the source corpus; training revisits that corpus. Dev and test data use the official independent splits. The tokenizer used during training is included in this repository.

Load with Transformers

Tested with Transformers 4.51.3 and PyTorch 2.7.x. The repository includes the model implementation, loaded with trust_remote_code=True.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ypwhere/attnres4babylm2026-10m-l16-dynamic-s1338"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, trust_remote_code=True
).eval()

inputs = tokenizer("The child reads a book", return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits
next_token = logits[0, -1].argmax().item()
print(tokenizer.decode([next_token]))

Keep inputs within 512 tokens. The implementation supports causal scoring and generation without an efficient KV cache. This is a pretrained language model rather than an instruction-tuned chat model.

Evaluation

The paper uses the BabyLM evaluator at commit `3d57ddc`. Comparisons across evaluator versions may differ because of changes to datasets or scoring. The accompanying model collection includes the paired architectures and seeds used for those comparisons.

Verify downloaded files

SHA256SUMS lists one SHA256 digest and its exact filename per line. The weights are stored in model.safetensors; the entry for that filename checks the downloaded weights. Other entries check the configuration, tokenizer, model code, and license files individually. From the downloaded model directory, run:

sha256sum -c SHA256SUMS       # Linux
# macOS: shasum -a 256 -c SHA256SUMS

An OK result for each file confirms that it matches this model package. Download all files from the same repository revision when checking the hashes.

License

Model weights and accompanying code are released under the MIT license. The model implementation builds on nanoGPT.