pocharlies/Qwen3-Coder-30B-A3B-NVFP4-EAGLE3-DGXSpark

Qwen3-Coder-30B-A3B-Instruct — NVFP4 + int4 lm_head + EAGLE3, tuned for DGX Spark (GB10)

1.90× faster decode on code editing than the stock NVFP4 checkpoint, on a single NVIDIA DGX Spark (GB10, 128 GB unified memory, ~273 GB/s).

Measured, not estimated: 72.9 → 138.7 tok/s single-stream decode on a code-edit workload.

This repo ships everything needed to reproduce that number: the quantized weights, the speculative-decoding draft head, the exact vLLM flags, and the deployment manifest. See Why this repo is self-contained — that section exists because of a real failure mode we hit.

What is in here

Measured performance

Single-stream decode, steady state (prefill excluded from the denominator). Runs where any other process was using the GPU were discarded — the node exposes 2 time-sliced GPU replicas over 1 physical GB10, so a neighbouring pod silently halves your throughput.

Acceptance rate with EAGLE3 k=3:

Read this before enabling EAGLE3

EAGLE3 makes free-form generation 26 % SLOWER (71.4 → 52.8 tok/s). At 15 % acceptance you pay for the draft forward and get nothing back. The head was trained on code (OpenCoder-LLM/opc-sft-stage1), so it is excellent at code editing and poor at prose.

Enable it if your traffic is agentic coding. Do not enable it for a general chat endpoint.

ngram / prompt-lookup is a trap here

We measured it: 80 % acceptance, 3.39 tokens/step — and 45 % SLOWER than no speculation at all (72.9 → 39.9 tok/s). The reason is in vLLM's source: NgramProposer sets num_numba_thread_available = min(1, cpu_count // 2), hard-capped at 1, so the KMP search runs single-threaded on CPU at every decode step. EAGLE3 drafts on the GPU, which is why it converts its acceptance into real throughput and ngram does not.

How to serve it

Container image used for every number above:

vllm 0.23.1rc1.dev301+g04c2a8dea  (nightly 04c2a8d), arm64/sm121
torch 2.11.0+cu130 · transformers 5.12.1 · FlashInfer

vllm serve /path/to/this/repo \
  --served-model-name qwen3-coder-30b-a3b-nvfp4 \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --max-num-seqs 4 \
  --max-num-batched-tokens 32768 \
  --gpu-memory-utilization 0.42 \
  --kv-cache-memory-bytes 21474836480 \
  --attention-backend flashinfer \
  --speculative-config '{"method":"eagle3","model":"/path/to/this/repo/eagle3-head","num_speculative_tokens":3}' \
  --port 8888 --host 0.0.0.0

Notes that cost us time, so they are written down:

• —262144 is native. max_position_embeddings=262144, rope_scaling=null. No YaRN needed, and VLLM_ALLOW_LONG_MAX_MODEL_LEN is deliberately NOT set so a real length error stays loud.
• —The EAGLE3 head adds one layer on top of the target's 48, so it needs ~2 % more KV cache than the same context without speculation. At 262144/fp8 that is 12.25 GiB, not 12.0. A budget that fits without the drafter will fail with it.
• —No `--quantization` flag. vLLM autodetects compressed-tensors from config.json.
• —No `--reasoning-parser`. The Instruct variant never emits <think>.
• —The lm_head shard loads through MarlinLinearKernel (CompressedTensorsWNA16).

The EAGLE3 head

eagle3-head/ is lmsys/SGLang-EAGLE3-Qwen3-Coder-30B-A3B-Instruct-SpecForge (MIT) with one change: max_position_embeddings 2048 → 262144.

That change is not cosmetic. vLLM builds the draft's rotary cache with torch.arange(max_position_embeddings), so with the original config every position ≥2048 indexes out of bounds. RoPE is analytic, so extending the cache just computes more rows at the same theta.

Known, unverified: the head declares rope_theta=1e6 while the target declares 1e7. It works and accepts at 96 % on code edits; whether matching them would improve long-context acceptance (currently 33 %) has not been tested.

How the int4 lm_head was made

The stock NVFP4 checkpoint leaves lm_head in BF16 (it is listed in quantization_config.ignore). At vocab 151936 × hidden 2048 that is 622 MB read per decoded token. Decode on GB10 is memory-bandwidth bound, so that is pure tax.

Quantized to int4/group-32 it becomes 175 MB — 447 MB less per token.

lm_head.weight_packed  int32    [151936, 256]   # 8 int4 values per int32
lm_head.weight_scale   bfloat16 [151936,  64]   # 2048/32 = 64 groups
lm_head.weight_shape   int64    [2]

Symmetric, no zero-point, values in [-8, 7]. Quantization error: L2 relative 9.19 %, max abs 0.02344.

Cost: 79 seconds, no GPU required. Measured gain: +5.7 % on top of EAGLE3 (less than the ~15-20 % the bandwidth math predicts, because with speculation the bottleneck shifts — the draft head carries its own lm_head).

Full quantization from BF16 was evaluated and rejected: field-by-field, the public NVFP4 checkpoint already uses an equivalent observer, the same effective scale_dtype, and a code calibration set. Reproducing it would be 4-8 GPU-hours for no measurable gain. The lm_head is the only material difference.

Why this repo is self-contained

We tried to reproduce a published 154 tok/s figure for a different model from its public repo and could not. The reason turned out to be that the public repo shipped only the base weights — the speculative-decoding module that produced the speedup, the split lm_head shard, and the model.safetensors.index.json that maps them were never uploaded. Anyone downloading it gets a checkpoint with no speculation and roughly a third of the tokens per forward pass, with nothing in the repo to indicate why.

So this repo ships the draft head, the index, the exact flags, the container build, and the measurement methodology including what we discarded and why. If a number here does not reproduce, that is a bug and we want to know.

Reproducing the measurements

The benchmark harness gates every run on /metrics and throws away any sample taken while another request — or another pod on the same time-sliced GPU — was active. Without that gate the same config measures anywhere from 27 to 139 tok/s, and the contaminated numbers look perfectly plausible.

State the workload with any tok/s claim. On this model the same config spans 52.8 → 138.7 tok/s depending only on whether the task is prose or code editing.

License

Apache-2.0, inherited from Qwen/Qwen3-Coder-30B-A3B-Instruct. eagle3-head/ is MIT, from lmsys.

Full benchmark campaign

Speed, tool-calling and agentic results for this model alongside three others on the same DGX Spark — including two optimizations that were measured and rejected — with all raw evidence:

https://huggingface.co/datasets/pocharlies/dgx-spark-moe-benchmarks