LLM-OS-Models/gdn2-370m-fineweb-edu-100b

GDN-2 370M (FineWeb-Edu 100B)

A pure-recurrent linear-attention language model trained from scratch on FineWeb-Edu. Architecture: Gated DeltaNet 2 (GDN-2) — the recurrence of Gated DeltaNet with two channel-wise gates.

This repository publishes the checkpoints produced by the campaign described in `docs/LMR_FULL_GUIDE_KO.md`. A new checkpoint is uploaded roughly every 5 B trained tokens.

1. What is GDN-2?

GDN-2 (Gated DeltaNet 2) is a pure-recurrent token mixer: there is no softmax attention, no sliding-window attention, and no Transformer block in the critical path. Every layer is a learned linear-recurrent state update.

Compared to its predecessor Gated DeltaNet (KDA), GDN-2 replaces the single scalar write/erase gate with two channel-wise gates:

$$ St \;=\; \bigl(I - kt (bt \odot kt)^{\!\top}\bigr)\,\mathrm{Diag}(\exp(gt))\,S{t-1} \;+\; kt (wt \odot v_t)^{\!\top} $$

• —$bt \in \mathbb{R}^{dk}$ — channel-wise erase gate (replaces KDA's scalar $\beta_t$)
• —$wt \in \mathbb{R}^{dv}$ — channel-wise write gate (new in GDN-2)
• —$g_t$ — output silu-gate (same as Gated DeltaNet)

Setting $bt = \betat\mathbf{1}$ and $wt = \betat\mathbf{1}$ recovers KDA exactly, so GDN-2 is a strict generalisation.

Why this matters for long-context: the recurrent state $St$ is $O(dk \cdot d_v)$ per head — constant in sequence length. Training and inference scale linearly with tokens, not quadratically like softmax attention.

2. Model configuration

name              = "gdn2_370M"
block_size        = 4096          # training context length
vocab_size        = 32000         # TinyLlama tokenizer
n_layer           = 16
n_head            = 8
n_embd            = 1024
head_dim          = 128
intermediate_size = 2048          # LLaMAMLP expansion
gdn2_per_layer    = 1             # 1 = pure recurrent, no SWA fallback
local_window      = 2048          # unused when gdn2_per_layer=1
rotary_percentage = 1.0
norm              = FusedRMSNorm (eps=1e-5)
mlp               = LLaMAMLP
parallel_residual = False
mamba_init        = True

The recurrent state per head is $dk \times dv = 128 \times 128 = 16{,}384$ floats. Across 8 heads and 16 layers this is 2.1 M recurrent state floats, designed to match Mamba-370M's recurrent-state budget.

3. Training recipe

Measured throughput on 8 × H200: 72.7 K tokens / sec / GPU (≈ 580 K tokens / sec aggregate). Wall-clock estimate end-to-end: ≈ 41 hours.

The exact launch script is checked in at `off/GatedDeltaNet-2/scripts/pretrain_gdn2_370m_fineweb_edu_100bt.sh`.

4. Live training status

This model is mid-training. New checkpoints appear here every ~5 B tokens. The latest live status is in `docs/OVERNIGHT_LIVE_STATUS_KO.md`.

Checkpoint naming gotcha (will be cleaned up post-run): the milestone file checkpoint-1B-model-ckpt.pth actually contains the 5 B-token state. The "1B" suffix is the milestone index (first 5 B milestone), not the token count. Subsequent milestones will be named checkpoint-2B-…, checkpoint-3B-…, etc. The README will be updated to clarify after the run completes.

5. How to load

The checkpoint is a raw PyTorch state dict in the layout used by lit_gpt.model.GPT configured with gdn2_370M. The repo also mirrors the training code (the lit_gpt/ package from off/GatedDeltaNet-2/).

import torch
from lit_gpt.config import config_from_name
from lit_gpt.model import GPT

ckpt = torch.load("checkpoint-1B-model-ckpt.pth", map_location="cpu")
# top-level key is "model" — the inner state dict
state = ckpt["model"] if "model" in ckpt else ckpt

cfg = config_from_name("gdn2_370M")
model = GPT(cfg)
model.load_state_dict(state, strict=True)
model.eval()

To run a quick continuation / generation, see the `off/GatedDeltaNet-2/` subproject — the same lit_gpt package is used for both training and inference.

6. Intended use

This model is released for research purposes only.

Appropriate uses:

• —Studying the GDN-2 recurrence and comparing against other linear / recurrent architectures (Mamba, RWKV, Gated DeltaNet, RetNet, Lightning Attention, …).
• —Long-context retrieval and associative-recall experiments where the $O(N)$ training cost matters.
• —Component-level ablations (gate design, head count, recurrent-state size).

Inappropriate uses:

• —Production deployment. The model is small (370 M), mid-training, and instruction-following has not been taught.
• —Downstream safety-critical tasks.
• —Anything requiring benchmark numbers we have not yet published. Wait for post-training evaluation.

7. Limitations (as of latest checkpoint)

• —Mid-training. Loss is still decreasing; downstream metrics will move.
• —Scale. 370 M parameters and a 4 K training context — small by modern standards. We chose this scale deliberately to match Mamba-370M and to fit a 36-hour campaign budget.
• —No instruction tuning. Outputs are raw next-token completions.
• —English-only training data (FineWeb-Edu is English academic web).
• —No benchmark numbers yet. HellaSwag / ARC / MMLU / RULER will be run on the final 100 B checkpoint and added here.

8. Evaluation plan (post-training)

Once the 100 B-token checkpoint lands we will run:

Results will be appended to this card and to `docs/LMR_PUBLIC_BENCHMARK_SUMMARY_KO.md`.

9. Citation

The GDN-2 architecture was introduced by NVIDIA in 2026. Please cite the upstream GDN-2 paper for the architecture itself.

For this specific checkpoint:

@misc{gdn2_370m_fineweb_edu_100b,
  title  = {GDN-2 370M trained on FineWeb-Edu 100B tokens},
  author = {LLM-OS-Models},
  year   = {2026},
  url    = {https://huggingface.co/LLM-OS-Models/gdn2-370m-fineweb-edu-100b},
  note   = {Work in progress; checkpoints published every 5B tokens}
}

10. Acknowledgements

• —The GDN-2 architecture and Triton kernels are from the Gated DeltaNet 2 authors (NVIDIA). This repo only trains their architecture.
• —Training data: HuggingFaceFW/fineweb-edu (sample/100BT slice).
• —Compute: 8 × NVIDIA H200 143 GB.
• —Tracking + live status infrastructure: the long-gdn campaign harness.