augustinian-babylm/token-embeddings

token-embeddings

Per-token visual embedding tables for the Augustinian BabyLM project: [V, 768] float32 matrices used to initialize the input embedding matrix of a DeBERTa-v3-base masked LM before text training.

Organized as <encoder>/<vocab>/, for encoder in dinov3 / sam / ibot and vocab in 50k / 75k / 100k. Each directory holds E_init.safetensors (the table) and a seeded_mask marking which rows carry visual information, roughly 24-38% of rows depending on vocabulary size.

Unseeded rows are zeros and must be overwritten with the model's own random initialization at load time, not used as-is.

Built by averaging the region features in `augustinian-babylm/region-embeddings` over every region a word labels, then mean-centering, L2-normalizing, and scaling to the model's initializer standard deviation. The published tables use all-subword attribution (--no-seed_last_subword); rebuild with the same flag for comparability.

Part of https://github.com/bylinina/augustinian_babylm. Paper: https://openreview.net/forum?id=B4TD4XdlwF.

Citation

@inproceedings{bylinina2026augustinian,
  title     = {Augustinian BabyLM: What Ostensive Definition Can and Cannot
               Teach a Small Language Model},
  author    = {Bylinina, Lisa},
  booktitle = {Proceedings of the BabyLM Workshop},
  year      = {2026},
  url       = {https://openreview.net/forum?id=B4TD4XdlwF}
}