Inferact/Kimi-K3-DSpark-Block5

Kimi-K3 DSpark — Block 5

A DSpark draft model for speculative decoding with `moonshotai/Kimi-K3`. Three full-causal MLA layers, conditioned on hidden states extracted from three of the target's layers and trained at block_size=5: one forward pass drafts the full five-token block.

Training

The draft completed a four-epoch run with **speculators**, the speculative-decoding library from the vLLM project, against Kimi-K3 hidden states streamed live from vLLM.

Acceptance

Evaluated against Kimi-K3 on 8×B300 (TP8), drafting 5 tokens with probabilistic sampling and block rejection at temperature 1.0 and top-p 0.95. All 1,604 requests succeeded. The aggregate is computed from summed counters, not an average of arm rates.

The separate five-shot GSM8K accuracy check scored 96.51% exact match on all 1,319 examples with zero request errors.

The full 1M context window is supported through YaRN, evaluated on every row of the BEAM 100K, 500K, and 1M splits above. The fixed 1M acceptance corpus left-truncates oldest turns in 14 of 35 over-limit source conversations to keep prompt plus output within 1,048,576 tokens; its longest measured prompt is 1,039,871 tokens. Two successful completions were flagged as degenerate by the client (BEAM 500K conversation 4 and BEAM 1M conversation 18). Full counters, per-position rates, and quality flags are in benchmark_results.json.

Serving

vllm serve moonshotai/Kimi-K3 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --max-model-len 1048576 \
  --speculative-config '{
    "method": "dspark",
    "model": "Inferact/Kimi-K3-DSpark-Block5",
    "num_speculative_tokens": 5,
    "attention_backend": "FLASHINFER_MLA",
    "draft_sample_method": "probabilistic",
    "rejection_sample_method": "block"
  }'