mlnomad/yatnmn-softplus-ca-d12-chinchilla-261M-pytorch

YatNMN-Softplus + constant α=1 d=12 Chinchilla (261M) — PyTorch

A 261M-parameter nanochat-architecture GPT with YatNMN-Softplus MLP where α is fixed at 1 (non-learnable). This is the α ablation — testing whether the learnable per-layer scaling in YatNMN is load-bearing.

Result: learnable α is critical. Fixing α=1 costs +0.12 nats on C4 and the model generalizes much worse on wikitext (PPL 67 vs 40).

Ablation table (d=12, 261M, Chinchilla 20×, 3-seed mean)

Quick start

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "mlnomad/yatnmn-softplus-ca-d12-chinchilla-261M-pytorch",
    trust_remote_code=True, dtype=torch.float32,
).eval()
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")

ids = tokenizer("The meaning of life is", return_tensors="pt").input_ids
with torch.no_grad():
    out = model.generate(ids, max_new_tokens=50, do_sample=False,
                         use_cache=True, pad_token_id=tokenizer.eos_token_id or 0)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Config

Per-neuron (ff,) bias + softplusbias + learnableepsilon + constant_alpha=True (α=1 fixed).

Related

• —`mlnomad/yatnmn-softplus-d12-chinchilla-261M-pytorch` — with learnable α (loss 2.98, PPL 40)
• —`mlnomad/gelu-d12-chinchilla-261M-pytorch` — GELU baseline

License

Apache 2.0.