Modelpublic

LLM-OS-Models/gdn2-370m-fineweb-edu-5b

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face

0likes

Model Card

GDN-2 370M (FineWeb-Edu 5B) — vanilla baseline


Architecture	GatedDeltaNet-2 (Yang et al. 2025), 370M parameter variant
Parameters	370 M
Training data	FineWeb-Edu sample/100BT (5 B-token slice)
Tokenizer	TinyLlama v1.1 (vocab = 32 000)
Context length	4 096 (training)
Hardware	8 × NVIDIA H200 141 GB (FSDP)
License	Apache-2.0
Trained by	LLM-OS-Models · code at gyunggyung/long-gdn

1. Purpose

Vanilla GDN-2 370M baseline trained on the same 5B-token slice as dsc-370m-fineweb-edu-5b-v3, so the DSC vs vanilla comparison is apples-to-apples (same data, same compute, same hyperparameters).

2. RULER results (vs DSC v3 5B)

DSC v3 vs Vanilla 5B comparison is the 5B verdict that determines whether DSC is a useful architectural addition to GDN-2. Full results in docs/DSC_LIVE_STATUS_KO.md. This vanilla checkpoint is the floor that DSC improvement is measured against.

3. Known limitations

—5B tokens below Chinchilla optimal (~7.5B for 370M params)
—Long-context (≥16K) retrieval fails completely (vanilla linear attention state collapse) — the gap DSC is designed to close

4. Citation

@misc{gdn2-vanilla-5b-2026,
  author = {LLM-OS-Models},
  title  = {GDN-2 370M FineWeb-Edu 5B vanilla baseline},
  year   = {2026},
  url    = {https://huggingface.co/LLM-OS-Models/gdn2-370m-fineweb-edu-5b}
}