yashmahe2018/birwkv7-strict-small-babylm2026

BiRWKV-7 — BabyLM 2026 Strict-Small

Architecture: BiRWKV-7 (~28M parameters) — bidirectional RWKV-7 with dual-pass fused inference Task: BabyLM 2026 Strict-Small track — primary BiRWKV-7 submission (seed 42, step 18k) Training corpus: 14.87M tokens (BabyLM Strict-Small 10M-word corpus, 32K BPE) Inference mode: CLM-only forward pass (BabyLM official eval)

Results (BabyLM 2026 official evaluation)

Architecture

• —8 layers, hidden dim 384, 6 attention heads (head size 64), 32K BPE vocabulary
• —Bidirectional: forward RWKV-7 pass + reversed backward pass, fused with sigmoid gate
• —N-gram embeddings (MD5 hash mod 2048), additive n-gram fusion gate
• —Chunked WKV-7 kernel (chunk_size=64, diagonal-only approximation)
• —Auxiliary objectives: MLM + MTP (backward pass supervises forward representations)

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "yashmahe2018/birwkv7-strict-small-babylm2026",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
    "yashmahe2018/birwkv7-strict-small-babylm2026",
    trust_remote_code=True
)

Citation

@misc{maheshwari2026halved,
  title={Halved CLM Exposure Mitigates Late-Training Collapse in Small Recurrent Language Models},
  author={Maheshwari, Yash},
  year={2026},
  note={BabyLM 2026 submission}
}