inference-optimization/dflash-DeepSeek-V4-Flash-all-swa-muon-speculators-50k

DFlash Speculator for DeepSeek-V4-Flash (all-SWA, Muon, 50k)

A DFlash draft (speculator) model trained for speculative decoding with DeepSeek-V4-Flash as the verifier.

This is the best checkpoint selected by validation loss from a 50k-sample training run.

Key characteristics

• —Algorithm: DFlash
• —Verifier / target: deepseek-ai/DeepSeek-V4-Flash
• —Attention: Sliding-window attention (SWA) on all draft layers (sliding_window=2048)
• —Optimizer: Muon
• —Training data: 50k samples
• —Draft layers: 5 (hidden_size=4096, head_dim=256, hc_mult=4)
• —Speculative tokens: 7 (block_size 8)
• —Aux hidden-state layers: [3, 13, 23, 32, 42]
• —dtype: bfloat16

Validation metrics (best checkpoint)

Files

• —config.json / config.py — speculator config and model definition
• —model.safetensors — draft model weights
• —val_metrics.json — validation metrics for this checkpoint