LLM-OS-Models/gdn1-ruler-replay-repair-markerfix-1b

GDN1 RULER Replay Repair Markerfix 1B

This is a research checkpoint from the Long-GDN workspace.

Base Model

• —Base checkpoint: linear-moe-hub/Gated-Deltanet-1.3B
• —Architecture: Gated DeltaNet / linear recurrent attention
• —Base training data reported by the upstream model card: SlimPajama 100B-token sample
• —License inherited from upstream model card: Apache-2.0

Training Run

• —Local source path: runs/gdn1_ruler_replay_repair_markerfix_from_balanced200_1b_bs10_ft/checkpoint-382
• —Tokenizer source: runs/gdn1_32k_balanced_recovery_1b_bs10_ft/final
• —Training mode: full fine-tuning, no LoRA/adapter
• —Hardware target: 8x NVIDIA H200
• —Sequence length: 32768
• —Approximate additional token budget: 1.0B additional tokens
• —Manifest/config: configs/gdn1_memory_mix_ruler_replay_repair_markerfix.json

Intended Research Use

This checkpoint is intended for research on:

• —long-context associative recall
• —RULER/MQAR-style state tracking
• —recurrent-state contamination during long generation
• —Reference-State Reset with Rolling Replay, a GDN/RNN adaptation of the R-SWA idea

Usage

These checkpoints use the FLA Gated DeltaNet implementation. In the current Long-GDN environment, plain GatedDeltaNetForCausalLM.from_pretrained() can hit a Transformers 5.x tied-weight metadata issue. The robust path is to patch the FLA tied-weight metadata before loading.

Install/runtime requirements:

pip install torch transformers safetensors huggingface_hub
# plus an FLA package/source tree that provides:
#   fla.models.gated_deltanet.GatedDeltaNetForCausalLM

CPU Example

import torch
from transformers import AutoTokenizer
from fla.models.gated_deltanet import GatedDeltaNetForCausalLM

repo_id = "LLM-OS-Models/gdn1-ruler-replay-repair-markerfix-1b"

if isinstance(getattr(GatedDeltaNetForCausalLM, "_tied_weights_keys", None), list):
    GatedDeltaNetForCausalLM._tied_weights_keys = {
        "lm_head.weight": "model.embeddings.weight"
    }

tokenizer = AutoTokenizer.from_pretrained(repo_id, use_fast=True)
model = GatedDeltaNetForCausalLM.from_pretrained(repo_id, torch_dtype=torch.float32)
model.eval()

prompt = "A special magic number is 12345. What is the special magic number?"
inputs = tokenizer(prompt, return_tensors="pt")
with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Single-GPU bf16 Example

import torch
from transformers import AutoTokenizer
from fla.models.gated_deltanet import GatedDeltaNetForCausalLM

repo_id = "LLM-OS-Models/gdn1-ruler-replay-repair-markerfix-1b"

if isinstance(getattr(GatedDeltaNetForCausalLM, "_tied_weights_keys", None), list):
    GatedDeltaNetForCausalLM._tied_weights_keys = {
        "lm_head.weight": "model.embeddings.weight"
    }

tokenizer = AutoTokenizer.from_pretrained(repo_id, use_fast=True)
model = GatedDeltaNetForCausalLM.from_pretrained(repo_id, torch_dtype=torch.bfloat16).to("cuda")
model.eval()

prompt = "Reference facts:\n- key_alpha: value_123\n\nQuestion: key_alpha?\nAnswer:"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Long-GDN Local Loader

The project repository includes a more defensive loader at scripts/gdn1_common.py::load_gdn1_causal_lm. It handles the compatibility patch and older public-checkpoint key conversion used in local experiments.

from pathlib import Path
import torch
from transformers import AutoTokenizer
from scripts.gdn1_common import load_gdn1_causal_lm

repo_or_local_path = Path("path/to/downloaded/checkpoint")
tokenizer = AutoTokenizer.from_pretrained(repo_or_local_path, use_fast=True)
model = load_gdn1_causal_lm(repo_or_local_path, torch_dtype=torch.bfloat16).to("cuda")

Known Results

MQAR likelihood checkpoint-382: 1K 0.1094, 2K 0.0938, 4K 0.1406, 8K 0.0938, 16K 0.1250, 32K 0.1250, 64K 0.1250.

RULER NIAH Single-2 generate contains_all: 4K 0.9375, 8K 0.1875, 16K 0.0625. This improves the local 4K smoke over balanced checkpoint-200 but does not improve 8K/16K.

Reference-State Replay exact checkpoint-382: 2K normal 0.2422 -> replay 0.3750, 4K 0.3125 -> 0.4922, 8K 0.2604 -> 0.3958, 16K 0.3750 -> 0.6406, 32K 0.2812 -> 0.3750, 64K 0.1875 -> 0.8750.

Caveats

Research checkpoint only. This is not instruction-tuned or safety-aligned. It should not be described as solving long RULER input-side retrieval: 8K/16K RULER NIAH Single-2 did not improve over the prior baseline in the local smoke protocol.

Citation Context

Relevant background papers include Gated Delta Networks, Gated DeltaNet-2, Log-Linear Attention, and Unlimited OCR / R-SWA. This checkpoint does not implement a new architecture by itself; it is part of a checkpoint-preserving full fine-tuning and inference-control study.