Modelpublic

LLM-OS-Models/gdn2-1.3B-fineweb-edu-100b

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

1likes

Model Card

GDN-2 1.3B (FineWeb-Edu 100B)

A pure-recurrent linear-attention language model trained from scratch on FineWeb-Edu. Architecture: Gated DeltaNet-2 (GDN-2) — decoupling erase and write gates in linear attention.


Status	🟡 In-progress pretraining — checkpoints uploaded every ~5B tokens
Architecture	GDN-2 (pure recurrent, no sliding-window attention)
Parameters	1.3 B (1,302,638,112)
Training data	FineWeb-Edu sample/100BT (~100B English tokens, academic-focus web)
Tokenizer	TinyLlama v1.1 (vocab = 32,000)
Context length	4,096 (training)
Hardware	8 × NVIDIA H200 143GB (DDP)
License	Apache-2.0
Trained by	LLM-OS-Models · code at gyunggyung/long-gdn
Paper	Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention (arXiv:2605.22791)

A new checkpoint is uploaded roughly every 5B trained tokens (checkpoint-5B, checkpoint-10B, checkpoint-15B, …, checkpoint-100B).

Model configuration

python

name              = "gdn2_1.3B"
block_size        = 4096          # training context length
vocab_size        = 32000         # TinyLlama tokenizer
n_layer           = 18
n_head            = 18
n_embd            = 2304
head_dim          = 128
intermediate_size = 6208          # LLaMAMLP expansion
gdn2_per_layer    = 1             # 1 = pure recurrent, no SWA fallback
local_window      = 2048          # unused when gdn2_per_layer=1
rotary_percentage = 1.0
norm              = FusedRMSNorm (eps=1e-5)
mlp               = LLaMAMLP
parallel_residual = False
mamba_init        = True

Training recipe

Hyperparameter	Value
Corpus	FineWeb-Edu sample/100BT
Target tokens	100,000,000,000 (100B)
Optimizer	AdamW, β = (0.9, 0.95), weight_decay = 0.1
Gradient clip	1.0
Learning rate	3 × 10⁻⁴ (peak), cosine schedule
Warmup	1 × 10⁹ tokens
Micro-batch × GPU	4 sequences × 4,096 tokens
Global batch	1,024 sequences = 4,194,304 tokens / step
Data-parallel workers	8
Save interval	every 1,193 steps ≈ 5B tokens
Measured throughput	~34.5K tokens/sec/GPU (~276K tokens/sec aggregate)
Wall-clock estimate	~100 hours end-to-end

Launch script: `off/GatedDeltaNet-2/scripts/pretrain_gdn2_1.3B_fineweb_edu_100bt.sh`

How to load

The checkpoint is a raw PyTorch state dict in the layout used by lit_gpt.model.GPT configured with gdn2_1.3B.

python

import torch
from lit_gpt.config import Config
from lit_gpt.model import GPT

ckpt = torch.load("checkpoint-15B-model-ckpt.pth", map_location="cpu", weights_only=False)
state = ckpt["model"] if "model" in ckpt else ckpt

cfg = Config.from_name("gdn2_1.3B")
model = GPT(cfg)
model.load_state_dict(state, strict=True)
model.eval()

The lit_gpt/ package used for training and inference lives at `off/GatedDeltaNet-2/lit_gpt/` in the source repo.

Checkpoints

Each checkpoint-{N}B-model-ckpt.pth is a self-contained state dict (~17.4GB) at the {N}B-token training milestone. The latest is also mirrored as latest-model-ckpt.pth.

Checkpoint	Tokens	Status
checkpoint-5B	5B	✅ uploaded
checkpoint-10B	10B	✅ uploaded
checkpoint-15B	15B	✅ uploaded
checkpoint-20B … 95B	—	⏳ in progress
checkpoint-100B (final)	100B	⏳ target

Intended use

Released for research purposes only.

Appropriate: studying GDN-2 recurrence, comparing linear/recurrent architectures (Mamba-2, Gated DeltaNet, KDA, RetNet), long-context retrieval experiments, component-level ablations.

Inappropriate: production deployment, safety-critical tasks, downstream benchmarks (wait for post-training evaluation on the final 100B checkpoint).

Limitations

—Mid-training. Loss is still decreasing; downstream metrics will move.
—No instruction tuning. Outputs are raw next-token completions.
—English-only training data (FineWeb-Edu is English academic web).
—4K training context. Longer-context generalization not yet evaluated.