CoolFace
Modelpublic

pakphum/babylm2026-ipa

sourceHugging Facemitupdated 6mo agoView on Hugging Face
0likes18downloads
Model Card

babylm2026-ipa

LLaMA-style causal language model trained for BabyLM 2026 with IPA phonological feature augmentation.

Architecture

  • —Base: LLaMA-style transformer (RMSNorm, RoPE, SwiGLU FFN, weight-tied LM head)
  • —IPA variant: ipa_full — gated fusion of panphon IPA feature vectors at the embedding layer
  • —Hidden size: 512 | Layers: 6 | Heads: 8
  • —Vocab size: 32000 | Context: 512 tokens
  • —IPA dim: 24 (panphon binary articulatory features)
  • —Training split: 10M

IPA Fusion Variants

VariantDescription
baselineToken embeddings only
ipa_addembed + W·ipa
ipa_gateembed + σ(W_g·ipa) ⊙ (W·ipa)
ipa_fullGated fusion + MSE auxiliary reconstruction loss

Usage

python
from transformers import AutoConfig, AutoModelForCausalLM
import torch

config = AutoConfig.from_pretrained("pakphum/babylm2026-ipa", trust_remote_code=True)
model  = AutoModelForCausalLM.from_pretrained("pakphum/babylm2026-ipa", trust_remote_code=True)
model.eval()

# input_ids from your tokenizer
input_ids = torch.tensor([[1, 2, 3, 4]])

# Optional: supply IPA vectors [B, T, ipa_dim=24]. Zeros = no phonological signal.
ipa_vectors = torch.zeros(1, 4, config.ipa_dim)

with torch.no_grad():
    out = model(input_ids, ipa_vectors=ipa_vectors)
    # out.logits: [B, T, vocab_size]

Citation

BabyLM 2026 shared task.