albertge/llada-8b-full-sft-mix60k-4pass

LLaDA-8B full-sequence SFT on mix60k

This model is the full-sequence SFT control model described in the paper Register Tokens for Bounded-State Reasoning in Diffusion Language Models. Matched full-sequence-SFT control for the dLLM Registers paper. This model has no register or discrete-text carry channel.

Training

• —Initialization: GSAI-ML/LLaDA-8B-Base
• —Data: the exact 60K mix of OpenMathInstruct-2 and OpenCodeInstruct in albertge/mix60k-math-code-sft
• —Four independently noised diffusion-loss optimizer passes per example
• —Natural/dynamic sequence lengths, capped at 1024 completion tokens
• —One epoch, batch size 1 per rank, learning rate 2e-5, weight decay 0.1
• —Exact runner: SFT/sft_train_vanilla.py and modal_jobs/vanilla_sft.py

Recorded evaluation

For math, evaluation scores the first emitted answer. Under fresh 8 x 128, later chunks cannot read earlier generated text and carry no hidden state.

Repository: https://github.com/SprocketLab/dllm-registers