Modelpublic

ml-ryanlee/seedvar-looped-1e18-d896-seed45

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

0likes29downloads

Model Card

seedvar-looped-1e18-d896-seed45

Seed-variance run for Sparse Layers are Critical to Scaling Looped Language Models (arXiv:2605.09165), trained to measure run-to-run noise in the 1e18-FLOP benchmark numbers.


architecture	`looped`
d_model	896
effective layers	16
compute budget	1e18 FLOPs
training steps	41,346
parameters (stored)	168,048,384
peak LR	0.01
batch size	16
data-order seed	45
init seed	42
muP width_ratio	7.0 (d_base=128)

Important: what varies across these four seeds

Only the training data order. The initialization seed is fixed at 42 for all four runs, as is the validation-batch order. The spread across seeds 42-45 therefore measures data-order variance, which is a lower bound on full run-to-run variance — a study that also varied initialization would be expected to show equal or greater spread. Do not read these error bars as total training noise.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained(
    "ml-ryanlee/seedvar-looped-1e18-d896-seed45", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("gpt2")

Evaluated with OLMES 5-shot core_9mcqa::olmes. When evaluating, pass max_length=1024 — the RoPE buffer is sized to the 1024-token training context and longer sequences overflow it.