Datasetpublic

pocharlies/dgx-spark-moe-benchmarks

Four MoE models on a DGX Spark: speed, tool-calling, and what actually breaks Full measurement campaign on NVIDIA DGX Spark (GB10, 128 GB unified, ~273 GB/s), vLLM 0.23.1rc1.dev301+g04c2a8dea, arm64/sm121. Every number here is measured on this hardware, with the raw evidence included. The headline: on synthetic tool-calling benchmarks all four models score 91-95 %. In a real coding agent, three of them score 0-1 out of 14 and one scores 11 out of 14. If you pick a model from the… See the full description on the dataset page: https://huggingface.co/datasets/pocharlies/dgx-spark-moe-benchmarks.

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

0likes58downloads

Dataset Card

Four MoE models on a DGX Spark: speed, tool-calling, and what actually breaks

Full measurement campaign on NVIDIA DGX Spark (GB10, 128 GB unified, ~273 GB/s), vLLM 0.23.1rc1.dev301+g04c2a8dea, arm64/sm121. Every number here is measured on this hardware, with the raw evidence included.

The headline: on synthetic tool-calling benchmarks all four models score 91-95 %. In a real coding agent, three of them score 0-1 out of 14 and one scores 11 out of 14. If you pick a model from the synthetic score, you will pick wrong.

1. Speed (single-stream decode, contention-gated)

Steady-state decode; prefill excluded from the denominator. Any sample taken while another request — or another pod on the same time-sliced GPU — was in flight is discarded.

config	code_edit	free_gen	long_ctx (52K)
Ornith-1.0-35B NVFP4 + MTP k=3	154.5	104.5	149.4
Qwen3-Coder-30B NVFP4 + EAGLE3 + int4 lm_head	138.7	—	—
NVIDIA Qwen3.6-35B NVFP4 + MTP k=3	135.3	95.8	124.0
NVIDIA + FlashInfer autotune	135.1	93.7	121.8
NVIDIA + attention requantized to NVFP4	129.1	97.9	131.4
Qwen3-Coder, no speculation	72.9	71.4	42.3
Qwen3-Coder + ngram k=3	39.9	—	78.2

Speculative acceptance, measured per workload:

model	code_edit	long_ctx	free_gen
Ornith (MTP)	3.91 tok/step · 94-98 %	3.73 · 91 %	2.60 · 51-60 %
NVIDIA (MTP)	4.00 · 98-100 %	4.00 · 100 %	2.67 · 52-60 %
Qwen3-Coder (EAGLE3)	3.89 · 96 %	2.00 · 33 %	1.46 · 15 %

Speculative decoding is workload-dependent, not a constant multiplier. EAGLE3 gives Qwen3-Coder +80 % on code editing and −26 % on prose — at 15 % acceptance you pay for the draft and get nothing. Quote the workload with the number or the number means nothing.

2. Tool calling — synthetic

`tooling_matrix`, 370 deterministic cases, seed 45739, six phases.

phase	Ornith	Qwen3-Coder	dense-27B	NVIDIA Qwen3.6
decisions (200)	193	200	200	200
functional (50)	37	44	50	45
structured (20)	20	20	20	20
freeform (10)	7	8	3	7
chains (50)	50	50	50	50
negative (40)	33	30	30	30
TOTAL	340/370 · 91.9 %	352/370 · 95.1 %	353/370 · 95.4 %	352/370 · 95.1 %

Run-to-run variance on the same model and config was 1.4 % (Ornith scored 345 and 340 on two runs), so treat gaps under ~1.5 points as noise.

Form matrix: 500 tool schemas exposed per call, across 10 shapes (chat vs responses API, tool_choice auto/required/named, streaming on/off, strict, catalog narrowing).

	Ornith	Qwen3-Coder	dense-27B
passed	9/10	10/10	10/10

Only failure: Ornith on responses-auto.

3. Tool calling — a real agent

14 tasks driving [opencode](https://opencode.ai) 1.18.9 over a sandbox repo. Every task is graded by a script — a test that passes, a file containing X, a command exiting 0. No LLM judge. The graders were validated first: each one fails on the pristine repo and passes on a correct fix (14 ok, 0 bad).

model	tasks passed	emitted invented XML	produced a real tool call	median latency
Qwen3-Coder-30B	11/14	0/14	14/14	45 s
NVIDIA Qwen3.6-35B	1/14	5/14	1/14	5 s
Ornith-1.0-35B	0/14	9/14	1/14	5 s
dense-27B	0/14	5/14	0/14	7 s

What the three failures look like

Under opencode's ~9,300-token system prompt, the qwen3_5_moe models stop emitting API tool calls and start emitting invented XML in the response text:

Let me read the relevant files...

<think>

<read filePath="/…/config/settings.json" />

grep

<think>

<Grep>
  <pattern>LEGACY_RATE_TOKEN\s*=</pattern>
</Grep>

An unterminated <think> block, then a hallucinated tool syntax. With no </think>, the reasoning parser never fires, everything stays in content, and the tool parser sees nothing it recognises.

This is not a client bug. We proxied and logged the raw HTTP: opencode sends its 9 tools correctly with tool_choice: auto. And the same endpoint, given a short prompt with a well-formed tools array, answers perfectly:

finish_reason: tool_calls
tool_calls: [{"function": {"name": "grep", "arguments": "{\"pattern\": \"LEGACY_RATE_TOKEN\"}"}}]

The failure appears only under a large agent system prompt. Note the latencies: the models that fail do so in 5-7 s because they give up in one turn; the one that works takes 45 s because it actually does the work.

It is not an Ornith fine-tune regression either. Its base model (nvidia/Qwen3.6-35B-A3B) fails the same way, as does the 27B from the same family. qwen3_moe (Qwen3-Coder) is the only architecture here that holds its tool-call format under load.

Tried and rejected: switching to the qwen3_xml tool parser made it worse — 0/8 with a 900 s median latency (timeouts) versus 5 s.

4. Two optimizations that did not work

Both were plausible. Both were measured. Both are negative — and the negatives are the useful part of this repo.

FlashInfer autotuning: zero effect

vLLM logs an explicit warning:

No tuned config covers fp8_gemm … falling back to runner=CutlassFp8GemmRunner tactic=-1. This shape is outside the tuning bucket range … to avoid this perf cliff.

Persisting the autotune cache (VLLM_FLASHINFER_AUTOTUNE_CACHE_DIR) and running a warm pass first changed decode by 135.3 → 135.1 tok/s, and TTFT not at all.

Why: the shapes in the warning are 249 and 32,160 tokens — those are prefill. The decode gap was never there. Read which phase a warning refers to before optimizing for it.

Requantizing attention FP8 → NVFP4: catastrophic

NVIDIA's checkpoint is MIXED_PRECISION: MoE experts in NVFP4, attention in FP8. The byte math makes converting attention look very attractive:

per forward pass	GiB
routed experts (8 of 128 active)	1.05
shared expert	0.07
attention, FP8, always active	1.47 → 51 % of the total
lm_head	0.27
total	2.85

Attention is 5.5 % of the checkpoint but 51 % of the bytes read per token — the classic MoE decode profile: experts are sparse, attention is dense. Converting it should cut 26 % of per-forward bytes and lift the ceiling from 96 to 112 forwards/s.

We did it: 90 linear_attn tensors, FP8 → NVFP4 W4A16, using vLLM's own scaled_fp4_quant. Round-trip L2 error 9.25-9.41 %, linear_attn 0.94 → 0.53 GiB. Clean conversion, model loads and serves.

Result:

	base FP8	attention NVFP4
code_edit tok/s	135.3	129.1
MTP acceptance	98-100 %	81-87 %
tooling 370	352/370 · 95.1 %	120/370 · 32.4 %

phase	base	requantized	Δ
decisions	200/200	66/200	−134
functional	45/50	5/50	−40
structured	20/20	0/20	−20
chains	50/50	12/50	−38
freeform	7/10	7/10	0
negative	30/40	30/40	0

Slower AND 62.7 points worse. Look at the pattern: freeform and negative — where the model writes prose or abstains — are untouched. Everything requiring exact structured output collapses. A 9.3 % error in attention is invisible in prose and fatal to a tool call.

Two things worth taking away:

1.NVIDIA's FP8 attention is a deliberate, correct choice. Now measured rather than assumed.
2.MTP acceptance is a free quality probe. It fell from 98-100 % to 81-87 % before we ran any quality battery. In MTP the draft and the verifier share the model: degrade the model and the draft's predictions stop matching. If acceptance drops after a change, the change hurt the model — you do not need a benchmark to know something is wrong, only to measure how much.

5. How to read a tok/s number

Three traps, each of which cost us hours:

—State the workload. The same model and config measured 52.8 to 138.7 tok/s depending only on whether the task was prose or code editing.
—A neighbour on a time-sliced GPU halves your numbers silently. We measured one config at 138.7 and at 26.8 tok/s. The contaminated figure looked entirely plausible. Gate on /metrics and discard samples taken with num_requests_running > 0.
—Verify the config you think is running is running. We once concluded EAGLE3 "gave zero speedup" when a kubectl apply had silently reverted the flag to its manifest default. The tell is spec_decode_num_accepted_tokens_total — absent counters mean nothing is speculating. "On but useless" and "never enabled" are indistinguishable from throughput alone.

evidence/
  tool370-{ornith,qwen3coder,dense27b,nv-base,nv-attnfp4}.jsonl   370 cases each, per-case verdicts
  tooling500.jsonl                                                 30 runs: 3 models x 10 shapes
  {ornith,nvidia,sweep}-*.json                                     decode measurements, per-run
agentic/
  results-*.jsonl                                                  14 tasks x 4 models, script-graded
scripts/
  decode_speed.py            contention-gated decode measurement
  requant_attn_nvfp4.py      FP8 -> NVFP4 attention requantizer (with round-trip validation)
  agentic_suite/             14 tasks, runner, scorer, sandbox generator

—`pocharlies/Qwen3-Coder-30B-A3B-NVFP4-EAGLE3-DGXSpark` — the 138.7 tok/s build, complete and reproducible
—`pocharlies/Ornith-1.0-35B-NVFP4-MTP-graft` — the 154.5 tok/s build

Recommendation

For an agentic coding CLI, Qwen3-Coder-30B-A3B, despite being 10 % slower than Ornith and tied on synthetic tooling. It is the only one of the four that reliably emits tool calls under a real agent's system prompt, and 11/14 versus 0/14 dwarfs a 16 tok/s difference.

For non-agentic serving, Ornith is the fastest here.