Modelpublic

staticlabs/dlm-code0.6b-exp

sourceHugging Faceupdated 18d agoView on Hugging Face

0likes440downloads

Model Card

dlm-code0.6b-exp

Experimental masked-diffusion language model for code, from the thunder-fast project.

This is a discrete masked-diffusion (MDM) adaptation of a Qwen2-based pretrained code model (~0.5B). It is not an autoregressive next-token model: generation is a progressive un-masking loop under bidirectional attention, so a whole output window is filled in parallel over 24 diffusion steps.

Details


Backbone	Qwen2 (`model_type: qwen2`), 24 layers, hidden 896, 14 attention / 2 KV heads, GQA
Vocabulary	151,936 + `<M>` mask token (id 151665)
Diffusion	discrete `[MASK]`, 24-step progressive un-masking, entropy-based position confidence (`alg_temp` 0.6, `top_k` 500)
Context	window-based (256 tokens) — not left-to-right; supports infilling
Weights	bf16 (`model.safetensors`)

How to run

This checkpoint is intended to be driven by a diffusion decoder (bidirectional attention + the iterative un-masking loop), e.g. the thunder-fast runtime. Loading it as a plain causal Qwen2ForCausalLM will not reproduce diffusion generation.

# thunder-fast runtime (ggml engine) or the reference DiffusionLM
# e.g. runtime --model dlm-code0.6b-exp.gguf --prompt "..."

Status

Experimental / work-in-progress (.exp). It is a diffusion code-generation checkpoint published for the thunder-fast project; see the repo for training, the masking schedule, and evaluation details.