mredmond/Kimi-K2.7-Code-DFlash-kimi-ft-v3

Kimi-K2.7-Code-DFlash — Hex fine-tune (kimi-ft-v3)

A DFlash speculative-decoding draft model for moonshotai/Kimi-K2.7-Code (served as nvidia/Kimi-K2.7-Code-NVFP4), fine-tuned on 3,071 real agent-harness conversations from Hex's coding agent.

The stock NVIDIA draft is trained on generic text; Hex agent traffic is not generic — every conversation carries a large system prompt, notebook and schema context, and long tool-call exchanges. Fine-tuning the draft on that distribution makes it measurably better at predicting the target exactly where it is deployed. Speculative decoding is output-lossless: the draft affects speed only, never output quality.

Full details in [`tech-report.pdf`](./tech-report.pdf) (methodology, training curves, verified sweeps, and an EAGLE3 comparison).

Headline results

Measured deployed, against the stock draft, under a conversation-paired protocol (28–29 held-out conversations x 3–6 repeats, verified warm caches, single-container servers, stall-robust decode TPS):

The accept-length advantage holds at every concurrency level (+2–13% at c=2–8); the TPS speedup concentrates at low concurrency, as expected for speculative decoding. Against an off-the-shelf EAGLE3 draft for the same target (AQ-MedAI/Kimi-K2.7-Code-eagle3, comparable to the production Baseten deployment), this model is +106% on paired decode TPS at concurrency 1.

Serving (SGLang v0.5.14)

Serve the NVFP4 target with this repo as the speculative draft:

python -m sglang.launch_server \
  --model-path nvidia/Kimi-K2.7-Code-NVFP4 \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path mredmond/Kimi-K2.7-Code-DFlash-kimi-ft-v3 \
  --speculative-dflash-block-size 8 \
  --speculative-draft-attention-backend fa4 \
  --attention-backend trtllm_mla \
  --kv-cache-dtype bf16 \
  --moe-runner-backend flashinfer_trtllm \
  --mem-fraction-static 0.85 \
  --tool-call-parser kimi_k2 --reasoning-parser kimi_k2 \
  --trust-remote-code --tp 8

Notes (hard-won; see the report and specdec/serving/variants.py):

• —bf16 KV cache is required: the draft worker inherits the global --kv-cache-dtype and its fa4 kernel asserts bf16 q/k/v, which also rules out the fp8-only tokenspeed_mla target backend — use trtllm_mla.
• —Do not pass `--quantization` explicitly: the draft worker deep-copies the target's server args, and an explicit modelopt_fp4 makes SGLang try to quantize the (bf16, unquantized) draft on the fly. Left unset, the target auto-detects NVFP4 from its own hf_quant_config.json and the draft loads as plain bf16.
• —Block size 8 is this checkpoint's native block size — the geometry it was trained (and fine-tuned) at.

Training

The draft trains against the target's intermediate hidden states (6 auxiliary layers x 7,168 dims). Fine-tuning helps most at deep block positions — exactly what gates whole-block acceptance.

Files

• —model.safetensors — 69 tensors, bf16 (~7.0 GB). Same architecture as the base draft (6 layers, hidden 7168, YaRN rope); embed_tokens / lm_head are borrowed from the target at serving time, as with the stock checkpoint.
• —config.json — unchanged from the base draft.
• —tech-report.pdf — the kimi-ft-v3 technical report (July 2026).

Caveats

• —Fine-tuned for Hex agent-harness traffic; on generic text expect roughly stock-draft behavior, not gains.
• —Evaluations above use temperature 1.0 with ignore_eos on held-out conversations collected against the stock-draft server (if anything, a bias against the fine-tune).
• —Exported from training checkpoint iter_0000384 (step 383) of the kimi-ft-v3 run; provenance (input hashes, version pins) is recorded in the run manifest in the specdec repo.