Felipe97/llama-cpp-compiled
01.1k
1# Speculative Decoding2 3llama.cpp supports speculative decoding, a technique that can significantly accelerate token generation by predicting multiple tokens ahead of the main model.4 5[Speculative decoding](https://en.wikipedia.org/wiki/Transformer_(deep_learning)#Speculative_decoding) leverages the fact that computing n tokens in a batch (as in prompt processing) is more efficient than computing n sequentially (as in response generation). By generating draft tokens quickly and then verifying them with the target model in a single batch, this approach can achieve substantial speedups when the draft predictions are frequently correct.6 7## Implementations8 9The `llama-server` application supports several implementations of speculative decoding. An implementation with draft model can be mixed with an implementation without draft model.10 11### Draft Model (`draft`)12 13A much smaller model (called the _draft model_) generates drafts.14A draft model is the most used approach in speculative decoding.15 16### EAGLE-3 (`draft-eagle3`)17 18EAGLE-3 uses a small draft model that reads the target model's hidden states to predict the next tokens, so it19reaches higher acceptance than a standalone draft model of the same size. The draft is a one-layer transformer20trained for a specific target model; it shares the target model's tokenizer and, optionally, uses a reduced draft21vocabulary with its own `lm_head`, which is mapped back using a `d2t` table.22 23Convert the EAGLE-3 checkpoint with `--target-model-dir` so it inherits the target's tokenizer and the layer24indices to read. Both the SpecForge `LlamaForCausalLMEagle3` and the vLLM/AngelSlim `Eagle3LlamaForCausalLM`25checkpoint formats are supported (for example [`AngelSlim/Qwen3-4B_eagle3`](https://huggingface.co/AngelSlim/Qwen3-4B_eagle3)26for `Qwen/Qwen3-4B`):27 28```bash29python convert_hf_to_gguf.py AngelSlim/Qwen3-4B_eagle3 \30 --target-model-dir Qwen/Qwen3-4B --outtype bf16 --outfile Qwen3-4B-eagle3.gguf31 32llama-server -m Qwen3-4B.gguf -md Qwen3-4B-eagle3.gguf --spec-type draft-eagle333```34 35Supported EAGLE-3 draft models include:36 37- [yuhuili/EAGLE3-LLaMA3.1-Instruct-8B](https://huggingface.co/yuhuili/EAGLE3-LLaMA3.1-Instruct-8B)38- [yuhuili/EAGLE3-LLaMA3.3-Instruct-70B](https://huggingface.co/yuhuili/EAGLE3-LLaMA3.3-Instruct-70B)39- [RedHatAI/gemma-4-31B-it-speculator.eagle3](https://huggingface.co/RedHatAI/gemma-4-31B-it-speculator.eagle3)40- [RedHatAI/gemma-4-26B-A4B-it-speculator.eagle3](https://huggingface.co/RedHatAI/gemma-4-26B-A4B-it-speculator.eagle3)41- [Tengyunw/qwen3_8b_eagle3](https://huggingface.co/Tengyunw/qwen3_8b_eagle3)42- [Tengyunw/qwen3_30b_moe_eagle3](https://huggingface.co/Tengyunw/qwen3_30b_moe_eagle3)43- [AngelSlim/Qwen3-1.7B_eagle3](https://huggingface.co/AngelSlim/Qwen3-1.7B_eagle3)44- [AngelSlim/Qwen3-4B_eagle3](https://huggingface.co/AngelSlim/Qwen3-4B_eagle3)45- [AngelSlim/Qwen3-8B_eagle3](https://huggingface.co/AngelSlim/Qwen3-8B_eagle3)46- [AngelSlim/Qwen3-14B_eagle3](https://huggingface.co/AngelSlim/Qwen3-14B_eagle3)47- [AngelSlim/Qwen3-32B_eagle3](https://huggingface.co/AngelSlim/Qwen3-32B_eagle3)48- [AngelSlim/Qwen3-a3B_eagle3](https://huggingface.co/AngelSlim/Qwen3-a3B_eagle3)49- [RedHatAI/gpt-oss-20b-speculator.eagle3](https://huggingface.co/RedHatAI/gpt-oss-20b-speculator.eagle3)50- [lmsys/EAGLE3-gpt-oss-120b-bf16](https://huggingface.co/lmsys/EAGLE3-gpt-oss-120b-bf16)51- [nvidia/gpt-oss-120b-Eagle3-long-context](https://huggingface.co/nvidia/gpt-oss-120b-Eagle3-long-context)52 53For the full and up-to-date list of supported models, see #18039.54 55### DFlash (`draft-dflash`)56 57DFlash produces an entire block of draft tokens in a single forward pass (block diffusion) and58injects the target model's hidden states into the draft model's attention, instead of drafting one59token at a time. This keeps the draft model small while making drafting GPU-friendly. Unlike EAGLE-360(a single-layer autoregressive draft), the DFlash draft uses several transformer layers but emits a61whole block per draft step.62 63The draft is a small block-diffusion model trained for a specific target (for example64`z-lab/Qwen3-4B-DFlash` for `Qwen/Qwen3-4B`). Convert it with `--target-model-dir` so it inherits the65target's tokenizer and token embeddings:66 67```bash68python convert_hf_to_gguf.py z-lab/Qwen3-4B-DFlash \69 --target-model-dir Qwen/Qwen3-4B --outtype bf16 --outfile Qwen3-4B-DFlash.gguf70 71llama-server -m Qwen3-4B.gguf -md Qwen3-4B-DFlash.gguf \72 --spec-type draft-dflash --spec-draft-n-max 15 -fa on --jinja73```74 75`--spec-draft-n-max` is clamped to the draft model's trained block size.76 77See:78 79- #2210580 81### DSpark (`draft-dspark`)82 83DSpark extends DFlash with a semi-autoregressive _Markov head_: the draft still emits a whole84block per forward pass, but each block position's logits are biased by a low-rank term keyed on85the previous token, chained in-graph across the block. This keeps drafting at one decode per86block while recovering some of the left-to-right signal that pure block diffusion loses.87 88The draft is a small DeepSpec checkpoint trained for a specific target (for example89[`deepseek-ai/dspark_qwen3_4b_block7`](https://huggingface.co/deepseek-ai/dspark_qwen3_4b_block7)90for `Qwen/Qwen3-4B`). Convert it with `--target-model-dir` so it inherits the target's tokenizer91and token embeddings:92 93```bash94python convert_hf_to_gguf.py deepseek-ai/dspark_qwen3_4b_block7 \95 --target-model-dir Qwen/Qwen3-4B --outtype bf16 --outfile Qwen3-4B-DSpark.gguf96 97llama-server -m Qwen3-4B.gguf -md Qwen3-4B-DSpark.gguf \98 --spec-type draft-dspark --spec-draft-n-max 7 -fa on --jinja99```100 101`--spec-draft-n-max` is clamped to the draft model's trained block size.102 103`--spec-draft-conf-min P` truncates each drafted block at the first position whose predicted104acceptance (from the draft's confidence head, if present) falls below `P` (default 0 = disabled).105 106Currently only drafts with a Qwen3 backbone are supported; support for other backbones107(e.g. Gemma4) is planned.108 109DSpark drafts exported in the [speculators](https://github.com/vllm-project/speculators) format110(for example [`RedHatAI/gemma-4-31B-it-speculator.dspark`](https://huggingface.co/RedHatAI/gemma-4-31B-it-speculator.dspark))111convert the same way.112 113See:114 115- #25173116 117### n-gram Cache (`ngram-cache`)118 119An n-gram is a sequence of n tokens. The n-gram cache implementation maintains statistics about short n-gram sequences.120A draft is computed using probabilities derived from these statistics. External statistics can also be loaded from files for improved accuracy.121 122See:123 124- #5479, #6828, #6848125 126### n-gram Map (`ngram-simple`, `ngram-map-*`)127 128These implementations search the token history for patterns and use matching sequences as draft candidates.129They require no additional model but rely on patterns that have already appeared in the generated text.130An example to use this approach can be the rewriting of source code by a LLM.131 132#### n-gram Map (`ngram-simple`)133 134This implementation looks for the last n-gram in history that matches the current n-gram and creates a draft using the m tokens following the matched n-gram. It is the simplest self-speculative approach with minimal overhead.135 136```137llama-server [...] --spec-type ngram-simple --spec-draft-n-max 64138```139 140#### n-gram Map Key (`ngram-map-k`)141 142This implementation looks for the current n-gram of size n (called the _key_) in the token history. If the key n-gram is followed by the same m tokens (called the _mgram_) multiple times, it creates a draft using these m tokens. This approach requires a minimum number of occurrences (argument `--spec-ngram-map-k-min-hits`, default is 1) before generating drafts.143 144The number of accepted tokens is stored for each used n-gram.145 146**Example:**147```148llama-server [...] --spec-type ngram-map-k --spec-draft-n-max 64149```150 151#### n-gram Map Key-4-Values (`ngram-map-k4v`)152 153This experimental implementation looks for the current n-gram of size n (called the _key_) in the token history. For each key, up to four _values_ (n-grams of size m, called _mgrams_) are tracked. An internal statistic counts the occurrences of each mgram after the key n-gram. If one mgram is significantly more frequent than the others, it is used as the draft.154 155The number of accepted tokens is stored for each used n-gram.156 157**Example:** Server options to be used if there are a lot of longer repetitions.158```159llama-server [...] --spec-type ngram-map-k4v --spec-ngram-map-k4v-size-n 8 --spec-ngram-map-k4v-size-m 8 --spec-ngram-map-k4v-min-hits 2 --spec-draft-n-max 64160```161 162### n-gram Mod (`ngram-mod`)163 164Add basic ngram hasher for speculative decoding:165 166- For each ngram, compute a hash using LCG167- For each computed hash, store the next token168- During speculation, iteratively compute the rolling hash of the last n tokens and pick the next token from the storage169 170Some characteristics:171 172- Lightweight (~16 MB)173- Constant memory and complexity174- Can generate variable draft lengths (i.e. m is not fixed)175 176Currently, a single hash pool is shared across all server slots, so different requests can benefit from each other.177 178**Sample usage:**179 180```181# notes:182# - small `n` are not recommended183# - MoEs require long drafts184# - dense models: can reduce `--spec-ngram-mod-n-min` and `--spec-ngram-mod-n-max`185 186llama-server ... --spec-type ngram-mod --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64187```188 189Applications:190 191- Iterating over a block of text/code (e.g. in llama.vim)192- Reasoning models (when they have to repeat their thinking in the final answer)193- Summarization194 195Example Video:196 197- See #19164198 199### Differences between ngram-simple, ngram-map and ngram-mod200 201- ngram-simple looks for a previous matching n-gram and inserts the following m-gram.202- ngram-map-k looks for a previous matching n-gram and inserts the following m-gram but uses an internal hash-map of n-grams in the current context window.203- ngram-mod uses a hash pool which is shared across all server slots. The hash pool is a map from n-gram hash to the next token (not the next m-gram as in ngram-map).204 205## Command-Line Options206 207If a draft model is combined with a draftless decoding the draftless decoding has higher precedence.208 209### Backend Sampling210 211Use `--backend-sampling` to run supported target-model samplers on the model backend. Draft-model sampling uses the backend by default and can be controlled with `--spec-draft-backend-sampling` and `--no-spec-draft-backend-sampling`.212 213Unsupported samplers and device layouts fall back to CPU sampling. Tensor split mode does not support backend sampling. A fixed seed produces repeatable random draws, but stochastic CPU and backend sampling can still select different tokens because floating-point operations can differ between implementations and devices. Use greedy sampling when exact output matching is required.214 215### Synthetic Acceptance216 217`llama-server` and `llama-cli` can replace normal speculative verification with synthetic decisions for benchmarking. The generated output is not valid model output because accepted draft tokens do not have to match the target model.218 219Use exactly one of these options:220 221- `--spec-synth-rates P0,P1,...` sets unconditional per-position acceptance probabilities. Entry `i` is the probability that the first `i+1` draft tokens are all accepted. The number of entries must match the effective maximum draft length. Values must be finite, within `[0, 1]`, and monotonically non-increasing.222- `--spec-synth-len L` sets the target mean acceptance length, including the target token. For `K` maximum draft tokens, `L` must be within `[1, K+1]`. The server finds a constant conditional probability `p` such that `p + p^2 + ... + p^K = L - 1`, then uses unconditional rates `[p, p^2, ..., p^K]`.223 224### General Speculative Parameters225 226```227--spec-type [none|draft-simple|draft-eagle3|draft-dflash|draft-dspark|draft-mtp|ngram-cache|ngram-simple|ngram-map-k|ngram-map-k4v|ngram-mod]228 comma-separated list of types of speculative decoding to use229 (default: none)230 (env: LLAMA_ARG_SPEC_TYPE)231--spec-default use default speculative decoding config232 (enables ngram-mod)233```234 235### Draft Model Parameters236 237```238--spec-draft-model, -md, --model-draft FNAME239 draft model for speculative decoding (default: unused)240 (env: LLAMA_ARG_SPEC_DRAFT_MODEL)241--spec-draft-hf, -hfd, -hfrd, --hf-repo-draft <user>/<model>[:quant]242 HuggingFace repository for the draft model243 (env: LLAMA_ARG_SPEC_DRAFT_HF_REPO)244--spec-draft-n-max N245 number of tokens to draft for speculative decoding (default: 3)246 (env: LLAMA_ARG_SPEC_DRAFT_N_MAX)247--spec-draft-n-min N248 minimum number of draft tokens to use for speculative decoding (default: 0)249 (env: LLAMA_ARG_SPEC_DRAFT_N_MIN)250--spec-draft-p-split, --draft-p-split P251 speculative decoding split probability (default: 0.10)252 (env: LLAMA_ARG_SPEC_DRAFT_P_SPLIT)253--spec-draft-p-min, --draft-p-min P254 minimum speculative decoding probability (greedy) (default: 0.00)255 (env: LLAMA_ARG_SPEC_DRAFT_P_MIN)256--spec-draft-ngl, -ngld, --gpu-layers-draft, --n-gpu-layers-draft N257 max. number of draft model layers to store in VRAM, either an exact number, 'auto', or 'all' (default: auto)258 (env: LLAMA_ARG_N_GPU_LAYERS_DRAFT)259--spec-draft-device, -devd, --device-draft <dev1,dev2,..>260 comma-separated list of devices to use for offloading the draft model261 (use --list-devices to see available devices)262```263 264### Draft Model CPU Scheduling Parameters265 266```267--spec-draft-threads, -td, --threads-draft N268 number of CPU threads to use during generation269--spec-draft-threads-batch, -tbd, --threads-batch-draft N270 number of threads to use during batch and prompt processing (default: same as --threads-draft)271--spec-draft-cpu-mask, -Cd, --cpu-mask-draft M272 Draft model CPU affinity mask. Complements cpu-range-draft273--spec-draft-cpu-range, -Crd, --cpu-range-draft lo-hi274 Ranges of CPUs for affinity. Complements --cpu-mask-draft275--spec-draft-cpu-strict, --cpu-strict-draft <0|1>276 Use strict CPU placement for draft model (default: same as --cpu-strict)277--spec-draft-prio, --prio-draft N278 set draft process/thread priority : 0-normal, 1-medium, 2-high, 3-realtime279--spec-draft-poll, --poll-draft <0|1>280 Use polling to wait for draft model work (default: same as --poll)281--spec-draft-cpu-mask-batch, -Cbd, --cpu-mask-batch-draft M282 Draft model CPU affinity mask for batch. Complements cpu-range-batch-draft283--spec-draft-cpu-range-batch, -Crbd, --cpu-range-batch-draft lo-hi284 Ranges of CPUs for affinity for batch. Complements --cpu-mask-batch-draft285--spec-draft-cpu-strict-batch, --cpu-strict-batch-draft <0|1>286 Use strict CPU placement for draft model batch (default: --cpu-strict-draft)287--spec-draft-prio-batch, --prio-batch-draft N288 set draft process/thread priority for batch : 0-normal, 1-medium, 2-high, 3-realtime289--spec-draft-poll-batch, --poll-batch-draft <0|1>290 Use polling to wait for draft model work for batch (default: --poll-draft)291```292 293### Draft Model KV Cache and Tensor Override Parameters294 295```296--spec-draft-type-k, -ctkd, --cache-type-k-draft TYPE297 KV cache data type for K for the draft model298 allowed values: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1299 (env: LLAMA_ARG_SPEC_DRAFT_CACHE_TYPE_K)300--spec-draft-type-v, -ctvd, --cache-type-v-draft TYPE301 KV cache data type for V for the draft model302 allowed values: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1303 (env: LLAMA_ARG_SPEC_DRAFT_CACHE_TYPE_V)304--spec-draft-override-tensor, -otd, --override-tensor-draft <tensor name pattern>=<buffer type>,...305 override tensor buffer type for draft model306--spec-draft-cpu-moe, -cmoed, --cpu-moe-draft307 keep all Mixture of Experts (MoE) weights in the CPU for the draft model308 (env: LLAMA_ARG_SPEC_DRAFT_CPU_MOE)309--spec-draft-n-cpu-moe, --spec-draft-ncmoe, -ncmoed, --n-cpu-moe-draft N310 keep the MoE weights of the first N layers in the CPU for the draft model311 (env: LLAMA_ARG_SPEC_DRAFT_N_CPU_MOE)312```313 314### n-gram Mod Parameters315 316```317--spec-ngram-mod-n-match N318 ngram-mod lookup length (default: 24)319--spec-ngram-mod-n-min N320 minimum number of ngram tokens to use for ngram-based speculative decoding (default: 48)321--spec-ngram-mod-n-max N322 maximum number of ngram tokens to use for ngram-based speculative decoding (default: 64)323```324 325### n-gram Simple Parameters326 327```328--spec-ngram-simple-size-n N329 ngram size N for ngram-simple speculative decoding, length of lookup n-gram (default: 12)330--spec-ngram-simple-size-m N331 ngram size M for ngram-simple speculative decoding, length of draft m-gram (default: 48)332--spec-ngram-simple-min-hits N333 minimum hits for ngram-simple speculative decoding (default: 1)334```335 336### n-gram Map Key Parameters337 338```339--spec-ngram-map-k-size-n N340 ngram size N for ngram-map-k speculative decoding, length of lookup n-gram (default: 12)341--spec-ngram-map-k-size-m N342 ngram size M for ngram-map-k speculative decoding, length of draft m-gram (default: 48)343--spec-ngram-map-k-min-hits N344 minimum hits for ngram-map-k speculative decoding (default: 1)345```346 347### n-gram Map Key-4-Values Parameters348 349```350--spec-ngram-map-k4v-size-n N351 ngram size N for ngram-map-k4v speculative decoding, length of lookup n-gram (default: 12)352--spec-ngram-map-k4v-size-m N353 ngram size M for ngram-map-k4v speculative decoding, length of draft m-gram (default: 48)354--spec-ngram-map-k4v-min-hits N355 minimum hits for ngram-map-k4v speculative decoding (default: 1)356```357 358### `--spec-type TYPE`359 360Specifies a comma-separated list of speculative decoding types to use.361 362| Type | Description |363|------|-------------|364| `none` | No speculative decoding (default) |365| `draft-simple` | Use a simple draft model for speculation |366| `draft-eagle3` | Use an EAGLE-3 draft model that reads the target's hidden states |367| `draft-dflash` | Use a DFlash block-diffusion draft model that emits a block per step |368| `draft-dspark` | Use a DSpark draft model (DFlash backbone + semi-autoregressive Markov head) |369| `draft-mtp` | Use Multi Token Prediction (MTP) heads from the main model |370| `ngram-cache` | Use n-gram cache lookup |371| `ngram-simple` | Use simple n-gram pattern matching |372| `ngram-map-k` | Use n-gram pattern matching with n-gram-keys |373| `ngram-map-k4v` | Use n-gram pattern matching with n-gram-keys and up to four m-gram values (experimental) |374| `ngram-mod` | Use basic ngram hasher for speculative decoding with shared pool |375 376**Example:** Server-instance used to refactor source code.377```bash378./llama-server [...] --spec-type ngram-simple379```380 381**Example:** Multiple speculative implementations.382```bash383./llama-server [...] --spec-type ngram-mod,ngram-map-k4v384```385 386### `--spec-ngram-*-size-n N`387 388Sets the size N of the lookup n-gram for n-gram map based speculative decoding.389The n-gram size N determines how many tokens in a row to look back when searching for matching patterns.390 391Each n-gram implementation has its own parameter:392 393- `--spec-ngram-simple-size-n` for `ngram-simple`394- `--spec-ngram-map-k-size-n` for `ngram-map-k`395- `--spec-ngram-map-k4v-size-n` for `ngram-map-k4v`396- `--spec-ngram-mod-n-match` for `ngram-mod`397 398### `--spec-ngram-*-size-m M`399 400Sets the size M of the draft m-gram for n-gram map based speculative decoding.401The m-gram size determines how many tokens to draft when a match is found.402Larger values can provide more speedup but may reduce acceptance rate.403 404Each n-gram implementation has its own parameter:405 406- `--spec-ngram-simple-size-m` for `ngram-simple`407- `--spec-ngram-map-k-size-m` for `ngram-map-k`408- `--spec-ngram-map-k4v-size-m` for `ngram-map-k4v`409 410### `--spec-ngram-*-min-hits H`411 412This option defines how often a key has to appear in the token history to be used as a draft (default is 1).413 414Each n-gram implementation has its own parameter:415 416- `--spec-ngram-simple-min-hits` for `ngram-simple`417- `--spec-ngram-map-k-min-hits` for `ngram-map-k`418- `--spec-ngram-map-k4v-min-hits` for `ngram-map-k4v`419 420## Statistics421Each speculative decoding implementation prints statistics.422 423```424draft acceptance rate = 0.57576 ( 171 accepted / 297 generated)425statistics ngram_simple: #calls = 15, #gen drafts = 5, #acc drafts = 5, #gen tokens = 187, #acc tokens = 73426statistics draft: #calls = 10, #gen drafts = 10, #acc drafts = 10, #gen tokens = 110, #acc tokens = 98427```428 429```430draft acceptance rate = 0.70312 ( 90 accepted / 128 generated)431statistics ngram_mod: #calls = 810, #gen drafts = 15, #acc drafts = 15, #gen tokens = 960, #acc tokens = 730, dur(b,g,a) = 0.149, 0.347, 0.005 ms432```433 434```435statistics ngram_map_k: #calls(b,g,a) = 6 1690 26, #gen drafts = 26, #acc drafts = 26, #gen tokens = 1248, #acc tokens = 968, dur(b,g,a) = 2.234, 1.427, 0.016 ms436```437 438 439- `#calls(b,g,a)`: number of calls of begin (new prompt), generation and accumulation of this implementations440- `#gen drafts`: number of drafts generated by this implementation441- `#acc drafts`: number of drafts accepted (partially) by the main model442- `#gen tokens`: number of tokens generated by this implementation (including rejected tokens)443- `#acc tokens`: number of tokens accepted by the main model444- `dur(b,g,a): durations of begin (new prompt), generation and accumulation (process acceptance).445 446## Benchmarking447 448To measure the end-to-end effect of speculative decoding (throughput, latency, and draft acceptance) across diverse prompts, see the SPEED-Bench client in [tools/server/bench/speed-bench](../tools/server/bench/speed-bench/README.md).449It runs against a running `llama-server` and can compare a baseline run against a speculative-decoding run.450