stuchapin/Carnice-V2-27B-MTP-GGUF

Carnice-V2-27B + MTP — GGUF for Ampere/older NVIDIA

GGUF builds of `kai-os/Carnice-V2-27b` with the MTP (Multi-Token Prediction) speculative-decoding head grafted in, packaged for llama.cpp. The MTP tensors come from `sakamakismile/Carnice-V2-27b-NVFP4-TEXT-MTP` (whose NVFP4 build is Blackwell-only). This repo makes the same MTP-accelerated decoding available on Ampere / Ada / pre-Blackwell GPUs via GGUF.

⚠️ Requires unmerged llama.cpp PR #22673

These GGUFs only work with [llama.cpp PR #22673](https://github.com/ggml-org/llama.cpp/pull/22673), which adds MTP support and is still in draft as of 2026-05-11. Mainline llama.cpp will fail to load them.

git clone https://github.com/ggml-org/llama.cpp.git && cd llama.cpp
git fetch origin pull/22673/head:mtp-pr && git checkout mtp-pr
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=86
cmake --build build --target llama-server llama-quantize -j8

(CUDA_ARCHITECTURES=86 for RTX 3090; 89 for 4090/Ada, 90 for Hopper.)

Variants

*All variants keep the MTP block (`blk.64.`) at BF16.** Quantizing the MTP head didn't help speed and risked acceptance-rate degradation in testing.

Benchmarks — real agent workload

Measured on 2× RTX 3090 (Q80 and Q6K via tensor-split, others single-GPU), --spec-draft-n-max 1, -c 32768, KV q4_0, flash attention on, thinking mode enabled. Tested with 7 representative prompts from a Hermes-style agent workload:

1. 1.tool-call-meeting — emit JSON tool calls for a CRM workflow
2. 2.morning-brief-synthesis — 5-bullet summary from agent context
3. 3.concierge-permit-extract — structured extraction from permit text
4. 4.draft-touch-card — short-form professional writing
5. 5.lease-clause-explain — domain knowledge / explanation
6. 6.long-context-summarize — 30-lead CRM dump → top 3
7. 7.reasoning-lease-vs-buy — multi-step numerical reasoning

Aggregate

Per-prompt tok/s

MTP acceptance is steady 81–84% across all variants — quant choice didn't degrade the draft head's accuracy. Speed differences come almost entirely from the base model's memory bandwidth per token.

Reference: non-MTP vanilla decode on the same hardware sits at ~35 tok/s, so even the slowest MTP variant (Q80) is roughly even with vanilla; IQ4XS gives ~1.4× over the vanilla baseline on this same workload mix.

Method

The whole thing is reproducible end-to-end. Headline steps:

1. Source materials

• —Base weights: kai-os/Carnice-V2-27b (BF16 safetensors, 55 GB) — the Hermes-style SFT of Qwen3.6-27B.
• —MTP head: 15 mtp.* BF16 tensors (~850 MB) extracted from sakamakismile/Carnice-V2-27b-NVFP4-TEXT-MTP. sakamakismile kept these tensors unquantized inside their NVFP4 file (the surrounding model is FP8/U8), so no dequantization required — just lift them out with safetensors.safe_open.

2. Merge

A small Python script symlinks the base BF16 shards into a working directory, writes the 15 MTP tensors as a new model-mtp.safetensors, regenerates model.safetensors.index.json, and rewrites config.json to set language_model_only: true (Carnice's base config carries a vision tower we don't ship).

3. Convert to GGUF

convert_hf_to_gguf.py from llama.cpp PR #22673 has first-class support for Qwen3_5ForConditionalGeneration and automatically remaps the mtp.* tensors to llama.cpp's blk.64.nextn.* naming. Output is a single BF16 GGUF (~55 GB).

4. Quantize

llama-quantize from the same PR build, with --tensor-type "blk\.64\..*=bf16" to keep the MTP block at full precision. Five targets: Q80, Q6K, Q5KM, Q4KM, IQ4_XS.

5. The critical tuning find — --spec-draft-n-max 1

Default community guides recommend --spec-draft-n-max 3. For Carnice this is wrong by a wide margin. The MTP head has mtp_num_hidden_layers: 1 — it was trained to predict exactly one token ahead. Drafting more tokens recurses the same head autoregressively; errors cascade rapidly.

We swept --spec-draft-n-max values 1, 2, 3, 5, 7 on the same workload:

General rule: set --spec-draft-n-max = mtp_num_hidden_layers from the model's config.

6. What didn't help

We also exhaustively swept (each at --spec-draft-n-max 1):

• —MTP block quant (BF16 / Q80 / Q5K) — all within ~2% noise. Default to BF16 for safety.
• —KV cache type (q40 / q80) — within noise; f16 OOMs at 64K context.
• —Batch / ubatch sizes — within noise.
• —Thread counts (1, 4, 8, auto) — within noise (the workload is GPU-bound).
• —CUDA graph disable — flag absent in PR build.
• —`ik_llama.cpp` fork — ~40% slower for this architecture. Carnice has 48 linear-attention (Mamba2-style) layers; mainline+PR-22673's kernels for those are noticeably more optimized than ik_llama.cpp's. Stick with mainline+PR.

Running

Single-GPU (≤19 GB models)

./build/bin/llama-server \
  -m Carnice-V2-27B-IQ4_XS-mtp.gguf \
  --alias carnice-v2-mtp \
  -c 65536 --parallel 1 -ngl 99 \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --flash-attn on --no-context-shift \
  --jinja --reasoning-format deepseek --reasoning-budget 4096 \
  --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 \
  --batch-size 2048 --ubatch-size 1024 --metrics \
  --spec-type mtp --spec-draft-n-max 1 \
  --fit off \
  --host 127.0.0.1 --port 8001

Dual-GPU (Q6K / Q80)

Add --tensor-split 14,14 and CUDA_VISIBLE_DEVICES=0,1. Expect ~15% PCIe overhead vs a single ≥32GB card.

Verify MTP is active

In the server startup log:

set_mtp: MTP draft head registered (ctx_mtp=..., n_embd=5120)
slot   load_model: speculative decoding context initialized

In per-request response timings:

"timings": {
  "predicted_per_second": 47.6,
  "draft_n": 13,
  "draft_n_accepted": 13
}

draft_n_accepted / draft_n should be ≥ 70% — we routinely see 100% on tool-call workloads.

Build recipe (reproduce from source)

# Merge MTP tensors into BF16 base
import json
from pathlib import Path
from safetensors import safe_open
from safetensors.torch import save_file

BASE = Path("./kai-os--Carnice-V2-27b")
MTP_SRC = Path("./sakamakismile--Carnice-V2-27b-NVFP4-TEXT-MTP")
OUT = Path("./merged")
OUT.mkdir(exist_ok=True)

for f in BASE.iterdir():
    if f.name in ("model.safetensors.index.json", "config.json"): continue
    (OUT / f.name).symlink_to(f.resolve())

mtp = {}
with safe_open(MTP_SRC / "model.safetensors", framework="pt") as fp:
    for k in fp.keys():
        if k.startswith("mtp."):
            mtp[k] = fp.get_tensor(k)
save_file(mtp, str(OUT / "model-mtp.safetensors"))

with open(BASE / "model.safetensors.index.json") as f: idx = json.load(f)
for k in mtp: idx["weight_map"][k] = "model-mtp.safetensors"
with open(OUT / "model.safetensors.index.json", "w") as f: json.dump(idx, f)

with open(BASE / "config.json") as f: cfg = json.load(f)
cfg["language_model_only"] = True
for k in ("vision_config", "image_token_id", "video_token_id"):
    cfg.pop(k, None)
with open(OUT / "config.json", "w") as f: json.dump(cfg, f, indent=2)

# Convert merged HF checkpoint to BF16 GGUF
python convert_hf_to_gguf.py ./merged \
    --outfile Carnice-V2-27B-bf16-mtp.gguf --outtype bf16

# Quantize, preserving the MTP block at BF16
./build/bin/llama-quantize --tensor-type "blk\.64\..*=bf16" \
    Carnice-V2-27B-bf16-mtp.gguf Carnice-V2-27B-IQ4_XS-mtp.gguf IQ4_XS

Known issues

1. 1.PR #22673 trips `free(): invalid pointer` SIGABRT on graceful shutdown. Runtime is clean; the kernel reclaims the process. Cosmetic. Likely fixed before PR merges.
2. 2.Vision input + MTP crashes in PR #22673. These GGUFs are text-only (vision config stripped).
3. 3.Prefill is ~half-speed with MTP enabled vs non-MTP. Matters most for very long prompts; decode is where MTP wins back.

License & attribution

This work is Apache 2.0 (inherits from kai-os/Carnice-V2-27b). The underlying Qwen3.6 base is subject to the Tongyi Qianwen License — commercial use exceeding 700K MAU requires a separate license from Alibaba.

Credit chain

• —Qwen team (Alibaba) — Qwen3.6 base model
• —[kai-os](https://huggingface.co/kai-os) — Carnice-V2-27B SFT (Hermes-style agent training)
• —[sakamakismile](https://huggingface.co/sakamakismile) — MTP head graft + NVFP4 packaging; we extracted the MTP tensors from their build
• —am17an and llama.cpp contributors — MTP support via PR #22673
• —[stuchapin](https://huggingface.co/stuchapin) (this repo) — GGUF + MTP fusion for Ampere, --spec-draft-n-max 1 tuning finding, end-to-end agent-workload benchmarks

Changelog

• —2026-05-11 — Initial release. 5 GGUF variants (Q80, Q6K, Q5KM, Q4KM, IQ4_XS), all clean from BF16 source, MTP block preserved at BF16. llama.cpp PR #22673 commit 5d5f1b4. Real-workload benchmarks across 7 Hermes-style agent prompts.