nshah-fbcs/ilm-arfalm-childes

ILM and ArfaLM: Child-Scale Language Models on CHILDES

Three small language models trained on naturalistic English-UK child-directed speech (CHILDES) for research on architectural progress, optimizer contributions, and deployment on legacy computing hardware.

All three models are trained on the same corpus (250,757 conversation pairs from CHILDES Eng-UK, with 5 children held out for true generalization testing). The training data is released separately at nshah-fbcs/childes-engUK-conversational-pairs.

Models

ilmsgdchildes (era-authentic 1999 LSTM)

• —Architecture: 2-layer LSTM, 256 hidden units, weight-tied embedding
• —Parameters: 2,338,184
• —Vocabulary: Word-level, 5,000 tokens
• —Optimizer: SGD with momentum 0.9, ReduceLROnPlateau (1986)
• —Gradient clipping: Norm 0.25 (Pascanu, Mikolov, and Bengio, 2013)
• —Test perplexity (per-word, normalized): 56.62
• —Generalization gap: 1.01

This model uses only training techniques that were available in 1999. It exhibits the gradient explosion behaviour characteristic of vanilla SGD on LSTMs (visible in the training history JSON).

ilmadamwchildes (optimizer ablation)

• —Architecture: Identical to ilmsgdchildes (2-layer LSTM, 256 hidden)
• —Parameters: 2,338,184
• —Vocabulary: Word-level, 5,000 tokens
• —Optimizer: AdamW, cosine learning rate schedule
• —Test perplexity (per-word, normalized): 36.33
• —Generalization gap: 0.91 (best of any model)

This is the optimizer-only ablation that isolates the contribution of the modern optimizer change from the architecture change.

arfalm_childes (modern Transformer)

• —Architecture: 6-layer decoder-only Transformer, 6 attention heads, embedding 288
• —Parameters: 8,426,944
• —Vocabulary: Byte-pair encoding, 4,096 tokens (tokenizer included)
• —Optimizer: AdamW, cosine learning rate schedule
• —Test perplexity (per-word, normalized): 26.39
• —Generalization gap: 1.12

The modern reference model. Named for Arfa Karim Randhawa (1995-2012), the youngest Microsoft Certified Professional.

Decomposition (test perplexity, vocabulary-normalized)

The optimizer change contributes more to the total gain than the architectural change does on this corpus at this scale.

Loading Examples

Load ArfaLM (Transformer)

import torch
import torch.nn as nn
import torch.nn.functional as F
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer

# Download files
model_path = hf_hub_download("nshah-fbcs/ilm-arfalm-childes", "arfalm_childes/model.pt")
tok_path = hf_hub_download("nshah-fbcs/ilm-arfalm-childes", "arfalm_childes/tokenizer.json")

# Load checkpoint
ckpt = torch.load(model_path, map_location='cpu', weights_only=False)
args = ckpt['model_args']
print(f"Model: {args}")
print(f"Best val PPL (per-token, native): {ckpt['val_ppl']:.2f}")
print(f"Best test PPL (per-token, native): {ckpt['test_ppl']:.2f}")

# Reconstruct architecture and load weights (see paper Section 4.2)
# ...
tokenizer = Tokenizer.from_file(tok_path)

Load ILM-SGD (era-authentic LSTM)

from huggingface_hub import hf_hub_download
import torch, json

model_path = hf_hub_download("nshah-fbcs/ilm-arfalm-childes", "ilm_sgd_childes/model.pt")
vocab_path = hf_hub_download("nshah-fbcs/ilm-arfalm-childes", "ilm_sgd_childes/vocab.json")

ckpt = torch.load(model_path, map_location='cpu', weights_only=False)
with open(vocab_path) as f:
    word2idx = json.load(f)

# See paper Section 4.1 for architecture reconstruction

Citation

If you use these models, please cite:

Shah, N. A. (2026). ILM and ArfaLM: Child-Scale Language Models Trained on Natural Parent-Child Speech for Deployment on Sub-1GHz Legacy Hardware. Part 2: Deployment, Optimization, and Safety on Original Hardware (Version 1.0). Zenodo. https://doi.org/10.5281/zenodo.19952279

Shah, N. A. (2026). ILM and ArfaLM: Child-Scale Language Models Trained on Natural Parent-Child Speech for Deployment on Sub-1GHz Legacy Hardware. Part 2: Deployment, Optimization, and Safety on Original Hardware (Version 1.0). Zenodo. https://doi.org/10.5281/zenodo.19952279

BibTeX:

@misc{shah2026ilmarfalm_part2,
  author       = {Shah, Noman A.},
  title        = {{ILM and ArfaLM: Child-Scale Language Models Trained on Natural Parent-Child Speech for Deployment on Sub-1GHz Legacy Hardware. Part 2: Deployment, Optimization, and Safety on Original Hardware}},
  year         = {2026},
  publisher    = {Zenodo},
  version      = {1.0},
  doi          = {10.5281/zenodo.19952279},
  url          = {https://doi.org/10.5281/zenodo.19952279}
}

License

CC-BY-NC-4.0. Research and educational use only. The underlying CHILDES training data is governed by the TalkBank Code of Ethics.

Limitations

• —Domain narrowness: Trained only on English-UK child-directed speech. Will not generalize well to other languages, registers, or age groups.
• —No safety filtering applied during training. External guardrails are required for deployment to child users (deferred to Part 2 of the accompanying paper).
• —Per-token vs per-word perplexity: The .pt checkpoint metadata stores per-token perplexity (8.93 for ArfaLM, 47.48 for ILM-SGD). These are not directly comparable across models with different tokenization. The vocabulary-normalized per-word perplexities reported above are the appropriate cross-model comparison.

Dedications

ILM is dedicated to the memory of Imran Shah (1969 to 2019). ArfaLM is named for Arfa Karim Randhawa (1995 to 2012).