LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1

Ornith-1.0-35B — GGUF (llama.cpp, single-GPU tp=1)

Single-GPU llama.cpp GGUF package for `deepreinforce-ai/Ornith-1.0-35B`. The supported serving policy is tp=1 — one model copy per GPU. Multi-GPU tensor-parallel serving is intentionally out of scope for this version.

This release ships six body quants (Q3KM → Q80) plus an **integrated IQ4XS-MTP graft** that adds a native multi-token-prediction (MTP) draft head for low-concurrency speculative decode. Quant quality is measured against the upstream BF16 GGUF with a native llama.cpp next-token top-64 KL-divergence probe over 32 coding prompts.

TL;DR — pick an artifact

MTP graft note: active MTP is byte-for-byte identical to target-only at the next-token (API-visible) level, and gives ~1.3× single-stream decode. It is not bit-exact to target-only over long deterministic sequences — see MTP. Use target-only for strict reproduction, MTP for throughput on open-ended work.

Quantization & fidelity

Fidelity is KL(P_bf16 || P_candidate) over next-token top-64 distributions, 32 coding prompts, n_predict=1, temperature=-1, n_probs=64, keyed by token ID. Lower KLD and higher greedy top-1 agreement are better.

Primary fidelity table

[image]

Fidelity ladder — the integrated IQ4_XS-MTP graft (purple) lands between Q5_K_M and Q4_K_M, i.e. better next-token fidelity than its Q4_K_M neighbor.

† Top-1 vs mean KLD are different axes. Top-1 counts how often the single most-likely (argmax) token matches BF16's, over the 32 prompts; mean KLD measures divergence across the full top-64 distribution. Two quants can share a top-1 count yet differ a lot in KLD — e.g. Q3_K_M and IQ4_XS are both 27/32 on top-1, but Q3_K_M's distribution diverges ~2.5× more (0.3620 vs 0.1426 nats). Neither value is a typo.

• —The integrated IQ4_XS-MTP graft is 48.70% lower mean KLD than the plain IQ4_XS body (0.0731382442 vs 0.1425748206), and improves greedy top-1 from 27/32 to 29/32.
• —Active `draft-mtp` ≡ target-only at the next-token level: their top-64 next-token distributions are identical (KLD 0.0, 32/32) for the one-token API-visible probe.

Body-quant distribution detail (secondary)

Full per-quant distribution from the corrected top-64 probe (same 32 prompts). Mean / P50 / P95 / Max are explicit nats statistics of the per-prompt top-64 KLD; in this run the BF16 top-64 distribution captured mean probability mass 0.999965, so the top-64 approximation captured essentially all next-token mass.

The IQ4_XS-MTP graft uses the same probe on a different breakpoint context; its distribution detail (P50 0.0492811974 / P95 0.1593743642 / Max 0.3348258511) is in the MTP section. Source: benchmarks/kld-quant-vs-bf16-top64.md.

Serving performance

Aggregate decode throughput and p95 time-to-first-token across concurrency, llama.cpp tp=1, from scripts/bench_openai.py --stream --max-tokens 256 (0 failed requests per row). Q4KM is the speed pick; Q5KM nearly ties it at c16.

[image]

[image]

Profile note: the table above is a short-context profile — CTX_SIZE=8192 PARALLEL=16 exposes n_ctx = 512/slot, with ~23-token prompts and 256-token generations. Source: benchmarks/llamacpp-quant-benchmarks.md. For how prefill latency scales with prompt length, see long-context TTFT below.

Long-context TTFT (single stream)

p95 time-to-first-token vs prompt/context length, single GPU tp=1, single stream (CTX_SIZE=131072 PARALLEL=1, n_ctx/slot=131072, exact prompt token counts, 140 rows/quant, 0 failures).

[image]

p95 TTFT rises with prompt length (prefill cost) to ~6.3 s at 32k tokens for Q4KM; the IQ4XS body and IQ4XS-MTP graft prefill slightly faster at every length. Decode throughput falls from ~180–190 tok/s at 512 tokens to ~10 tok/s at 32k as the KV cache grows. Source: benchmarks/llamacpp-longctx-ttft.md.

Multi-token prediction (MTP)

The integrated IQ4_XS-MTP-graft-headQ6 artifact carries a native MTP draft head on the IQ4XS body. The release recommendation is **adaptive**: use MTP for low-concurrency / single-user requests and keep `LLAMASPECMAXDRAFTING_SLOTS=1` so saturated batches fall back to target-only throughput. The MTP catalog validates the published MTP profiles.

Next-token quality

Active MTP does not change the one-token, API-visible next-token distribution. Mean BF16 KLD is 0.0731382442 nats over the 32-prompt top-64 probe (P50 0.0492811974 / P95 0.1593743642 / Max 0.3348258511), top-1 29/32. The active draft-mtp vs target-only graft next-token KLD is 0.0 (32/32 identical).

Sequence-level behavior

A deterministic 8 prompt × 64 token probe (n_predict=64, temperature=-1, n_probs=64) compares the same GGUF running active native draft-mtp against target-only.

[image]

• —Fast active MTP gives ~1.35× client throughput (233.81 vs 172.57) and server decode 325.70 vs ~210, with 82.01% draft acceptance.
• —It matches 6/8 sequences exactly and 478/512 = 93.36% of token positions; the two non-exact sequences first diverge at positions 60 and 34.
• —The LLAMA_SPEC_VERIFY_SEQUENTIAL=1 variant is both slower (164.34 / 204.88) and matches less (5/8, 88.87%; divergences at 25, 43, 60) — i.e. the fast verifier is here both faster and more target-matching.

Note: MTP self-speculation is not bit-exact to target-only over long deterministic generations. For workloads that require exact target reproduction, run target-only; for throughput on open-ended work, MTP wins. MTP is a single-user / low-concurrency win — it is not faster on saturated batches, which is why the adaptive throttle falls back to target-only.

Sequence logprob comparability (API limitation)

llama.cpp omits candidate top_logprobs for most accepted speculative tokens, so a naive "sequence KLD" looks enormous — that is an API/logprob-availability artifact, not model quality. On the comparable subset where the emitted token also matches, KLD is ≈ 0.001 nats (near zero):

Full sequence KLD cannot be computed without changing the runtime to return target verifier logprobs for accepted speculative tokens.

Draft-head distillation provenance

The meaningful KL-style metric for a standalone MTP draft head is teacher KL against cached target hidden states, not standard base-model next-token KLD. The draft head was trained and measurably improved across checkpoints — teacher KL fell 1.92 → 1.50 → 0.76 and teacher argmax top-1 rose 0.54 → 0.69 → 0.89.

[image]

Teacher KL is measured against cached target hidden states (not base next-token KLD). These training artifacts live under `artifacts/mtp/` locally and are referenced by name as provenance — they are not committed in this HF repo.

Run with llama.cpp

Serve a body quant on GPU0 (tp=1):

QUANT=Q4_K_M PORT=8000 CTX_SIZE=8192 PARALLEL=16 REASONING=off \
  scripts/serve_llamacpp_gpu0.sh

Serve the integrated MTP graft for low-concurrency speculative decode (keep adaptive throttling so saturated batches fall back to target-only):

QUANT=IQ4_XS-MTP-graft-headQ6 PORT=8000 CTX_SIZE=8192 PARALLEL=1 \
  CACHE_RAM=0 REASONING=off LLAMA_SPEC_MAX_DRAFTING_SLOTS=1 \
  scripts/serve_llamacpp_gpu0.sh

REASONING=off is the default because the model otherwise spends simple coding prompts in reasoning_content before producing final content. See benchmarks/llamacpp-q4-reasoning-off-fix.md. Serving / quant / MTP profile catalogs: configs/serving_profiles.yaml, configs/quant_artifacts.yaml, configs/mtp_profiles.yaml.

Benchmark environment

Hardware: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 97,887 MiB VRAM, single GPU (tp=1). Driver 580.159.03, CUDA runtime reported by driver 13.0.

Dedicated MTP KLD eval config (2026-06-28): local llama.cpp CUDA server, single RTX PRO 6000 Blackwell 96GB, tp=1, CTX_SIZE=8192, PARALLEL=1, CACHE_RAM=0, REASONING=off.

• —llama.cpp build/commit: 050ee92d04c2e1f639025786dea701c70e7d4204 (pinned for the long-context TTFT run; earlier KLD/throughput sweeps were not separately pinned).
• —Reasoning mode: all benchmarks in this card ran with REASONING=off.

Provenance & reproducibility

• —Figures: regenerate with `scripts/make_charts.py` → assets/02_fidelity_ladder.png, assets/03_throughput_tps.png, assets/04_ttft_p95.png, assets/05_mtp_draft_head_distill.png, assets/06_mtp_tps.png.
• —Serving throughput / TTFT: benchmarks/llamacpp-quant-benchmarks.md.
• —Body-quant fidelity: benchmarks/kld-quant-vs-bf16-top64.md.
• —Dedicated MTP KLD / sequence eval (2026-06-28): benchmarks/mtp-kld-eval-2026-06-28.md and benchmarks/mtp-dedicated-kld-evaluation.md, with raw evidence under benchmarks/raw/. Includes the sequence-level probe and its logprob-availability limitation (full sequence KLD is not computable from the current llama.cpp API response).
• —Adaptive MTP serving profile: benchmarks/llamacpp-iq4-xs-mtp-graft-adaptive-profile.md.
• —Long-context TTFT: benchmarks/llamacpp-longctx-ttft.md.