Felipe97/llama-cpp-compiled

1# LLaMA.cpp HTTP Server2 3Fast, lightweight, pure C/C++ HTTP server based on [httplib](https://github.com/yhirose/cpp-httplib), [nlohmann::json](https://github.com/nlohmann/json) and **llama.cpp**.4 5Set of LLM REST APIs and a web UI to interact with llama.cpp.6 7**Features:**8 * LLM inference of F16 and quantized models on GPU and CPU9 * [OpenAI API](https://github.com/openai/openai-openapi) compatible chat completions, responses, and embeddings routes10 * [Anthropic Messages API](https://docs.anthropic.com/en/api/messages) compatible chat completions11 * Reranking endpoint (https://github.com/ggml-org/llama.cpp/pull/9510)12 * Parallel decoding with multi-user support13 * Continuous batching14 * Multimodal ([documentation](../../docs/multimodal.md)) / with OpenAI-compatible API support15 * Monitoring endpoints16 * Schema-constrained JSON response format17 * Prefilling of assistant messages similar to the Claude API18 * [Function calling](../../docs/function-calling.md) / tool use for ~any model19 * Speculative decoding20 * Easy-to-use web UI21 22For the full list of features, please refer to [server's changelog](https://github.com/ggml-org/llama.cpp/issues/9291)23 24## Usage25 26<!-- HELP_START -->27 28<!-- IMPORTANT: The list below is auto-generated by llama-gen-docs; do NOT modify it manually -->29 30### Common params31 32| Argument | Explanation |33| -------- | ----------- |34| `-h, --help, --usage` | print usage and exit |35| `--version` | show version and build info |36| `-cl, --cache-list` | show list of models in cache |37| `--completion-bash` | print source-able bash completion script for llama.cpp |38| `-t, --threads N` | number of CPU threads to use during generation (default: -1)<br/>(env: LLAMA_ARG_THREADS) |39| `-tb, --threads-batch N` | number of threads to use during batch and prompt processing (default: same as --threads) |40| `-C, --cpu-mask M` | CPU affinity mask: arbitrarily long hex. Complements cpu-range (default: "") |41| `-Cr, --cpu-range lo-hi` | range of CPUs for affinity. Complements --cpu-mask |42| `--cpu-strict <0\|1>` | use strict CPU placement (default: 0) |43| `--prio N` | set process/thread priority : low(-1), normal(0), medium(1), high(2), realtime(3) (default: 0) |44| `--poll <0...100>` | use polling level to wait for work (0 - no polling, default: 50) |45| `-Cb, --cpu-mask-batch M` | CPU affinity mask: arbitrarily long hex. Complements cpu-range-batch (default: same as --cpu-mask) |46| `-Crb, --cpu-range-batch lo-hi` | ranges of CPUs for affinity. Complements --cpu-mask-batch |47| `--cpu-strict-batch <0\|1>` | use strict CPU placement (default: same as --cpu-strict) |48| `--prio-batch N` | set process/thread priority : 0-normal, 1-medium, 2-high, 3-realtime (default: 0) |49| `--poll-batch <0\|1>` | use polling to wait for work (default: same as --poll) |50| `-c, --ctx-size N` | size of the prompt context (default: 0, 0 = loaded from model)<br/>(env: LLAMA_ARG_CTX_SIZE) |51| `-n, --predict, --n-predict N` | number of tokens to predict (default: -1, -1 = infinity)<br/>(env: LLAMA_ARG_N_PREDICT) |52| `-b, --batch-size N` | logical maximum batch size (default: 2048)<br/>(env: LLAMA_ARG_BATCH) |53| `-ub, --ubatch-size N` | physical maximum batch size (default: 512)<br/>(env: LLAMA_ARG_UBATCH) |54| `--keep N` | number of tokens to keep from the initial prompt (default: 0, -1 = all) |55| `--swa-full` | use full-size SWA cache (default: false)<br/>[(more info)](https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)<br/>(env: LLAMA_ARG_SWA_FULL) |56| `-fa, --flash-attn [on\|off\|auto]` | set Flash Attention use ('on', 'off', or 'auto', default: 'auto')<br/>(env: LLAMA_ARG_FLASH_ATTN) |57| `--perf, --no-perf` | whether to enable internal libllama performance timings (default: false)<br/>(env: LLAMA_ARG_PERF) |58| `-e, --escape, --no-escape` | whether to process escapes sequences (\n, \r, \t, \', \", \\) (default: true) |59| `--rope-scaling {none,linear,yarn}` | RoPE frequency scaling method, defaults to linear unless specified by the model<br/>(env: LLAMA_ARG_ROPE_SCALING_TYPE) |60| `--rope-scale N` | RoPE context scaling factor, expands context by a factor of N<br/>(env: LLAMA_ARG_ROPE_SCALE) |61| `--rope-freq-base N` | RoPE base frequency, used by NTK-aware scaling (default: loaded from model)<br/>(env: LLAMA_ARG_ROPE_FREQ_BASE) |62| `--rope-freq-scale N` | RoPE frequency scaling factor, expands context by a factor of 1/N<br/>(env: LLAMA_ARG_ROPE_FREQ_SCALE) |63| `--yarn-orig-ctx N` | YaRN: original context size of model (default: 0 = model training context size)<br/>(env: LLAMA_ARG_YARN_ORIG_CTX) |64| `--yarn-ext-factor N` | YaRN: extrapolation mix factor (default: -1.00, 0.0 = full interpolation)<br/>(env: LLAMA_ARG_YARN_EXT_FACTOR) |65| `--yarn-attn-factor N` | YaRN: scale sqrt(t) or attention magnitude (default: -1.00)<br/>(env: LLAMA_ARG_YARN_ATTN_FACTOR) |66| `--yarn-beta-slow N` | YaRN: high correction dim or alpha (default: -1.00)<br/>(env: LLAMA_ARG_YARN_BETA_SLOW) |67| `--yarn-beta-fast N` | YaRN: low correction dim or beta (default: -1.00)<br/>(env: LLAMA_ARG_YARN_BETA_FAST) |68| `-kvo, --kv-offload, -nkvo, --no-kv-offload` | whether to enable KV cache offloading (default: enabled)<br/>(env: LLAMA_ARG_KV_OFFLOAD) |69| `--repack, -nr, --no-repack` | whether to enable weight repacking (default: enabled)<br/>(env: LLAMA_ARG_REPACK) |70| `--no-host` | bypass host buffer allowing extra buffers to be used<br/>(env: LLAMA_ARG_NO_HOST) |71| `-ctk, --cache-type-k TYPE` | KV cache data type for K<br/>allowed values: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1<br/>(default: f16)<br/>(env: LLAMA_ARG_CACHE_TYPE_K) |72| `-ctv, --cache-type-v TYPE` | KV cache data type for V<br/>allowed values: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1<br/>(default: f16)<br/>(env: LLAMA_ARG_CACHE_TYPE_V) |73| `-dt, --defrag-thold N` | KV cache defragmentation threshold (DEPRECATED)<br/>(env: LLAMA_ARG_DEFRAG_THOLD) |74| `--rpc SERVERS` | comma-separated list of RPC servers (host:port)<br/>(env: LLAMA_ARG_RPC) |75| `-lm, --load-mode MODE` | model loading mode (default: auto)<br/>- auto: mmap, unless a device does not support it<br/>- none: no special loading mode<br/>- mmap: memory-map model (if mmap disabled, slower load but may reduce pageouts if not using mlock)<br/>- mlock: force system to keep model in RAM rather than swapping or compressing<br/>- mmap+mlock: mmap + force system to keep model in RAM rather than swapping or compressing<br/>- dio: use DirectIO if available<br/><br/>(env: LLAMA_ARG_LOAD_MODE) |76| `-lzm, --lazy-mode MODE` | on-demand reading of certain tensors, for example per-layer embeddings (default: auto)<br/>- on: read the rows of such tensors from disk on demand instead of keeping them resident (requires mmap)<br/>- auto: on, but only for tensors larger than 4 GiB<br/>- off: always keep them resident<br/>(env: LLAMA_ARG_LAZY_MODE) |77| `--numa TYPE` | attempt optimizations that help on some NUMA systems<br/>- distribute: spread execution evenly over all nodes<br/>- isolate: only spawn threads on CPUs on the node that execution started on<br/>- numactl: use the CPU map provided by numactl<br/>if run without this previously, it is recommended to drop the system page cache before using this<br/>see https://github.com/ggml-org/llama.cpp/issues/1437<br/>(env: LLAMA_ARG_NUMA) |78| `-dev, --device <dev1,dev2,..>` | comma-separated list of devices to use for offloading (none = don't offload)<br/>use --list-devices to see a list of available devices<br/>(env: LLAMA_ARG_DEVICE) |79| `--list-devices` | print list of available devices and exit |80| `-ot, --override-tensor <tensor name pattern>=<buffer type>,...` | override tensor buffer type<br/>(env: LLAMA_ARG_OVERRIDE_TENSOR) |81| `-cmoe, --cpu-moe` | keep all Mixture of Experts (MoE) weights in the CPU<br/>(env: LLAMA_ARG_CPU_MOE) |82| `-ncmoe, --n-cpu-moe N` | keep the Mixture of Experts (MoE) weights of the first N layers in the CPU<br/>(env: LLAMA_ARG_N_CPU_MOE) |83| `-ncffn, --n-cpu-ffn N` | keep the dense FFN weights of the first N layers in the CPU<br/>(dense models; for MoE expert weights use --n-cpu-moe)<br/>(env: LLAMA_ARG_N_CPU_FFN) |84| `-ngl, --gpu-layers, --n-gpu-layers N` | max. number of layers to store in VRAM, either an exact number, 'auto', or 'all' (default: auto)<br/>(env: LLAMA_ARG_N_GPU_LAYERS) |85| `-sm, --split-mode {none,layer,row,tensor}` | how to split the model across multiple GPUs, one of:<br/>- none: use one GPU only<br/>- layer (default): split layers and KV across GPUs (pipelined)<br/>- row: split weight across GPUs by rows (parallelized)<br/>- tensor: split weights and KV across GPUs (parallelized, EXPERIMENTAL)<br/>(env: LLAMA_ARG_SPLIT_MODE) |86| `-ts, --tensor-split N0,N1,N2,...` | fraction of the model to offload to each GPU, comma-separated list of proportions, e.g. 3,1<br/>(env: LLAMA_ARG_TENSOR_SPLIT) |87| `-mg, --main-gpu INDEX` | the GPU to use for the model (with split-mode = none), or for intermediate results and KV (with split-mode = row) (default: 0)<br/>(env: LLAMA_ARG_MAIN_GPU) |88| `-fit, --fit [on\|off]` | whether to adjust unset arguments to fit in device memory ('on' or 'off', default: 'on')<br/>(env: LLAMA_ARG_FIT) |89| `-fitt, --fit-target MiB0,MiB1,MiB2,...` | target margin per device for --fit, comma-separated list of values, single value is broadcast across all devices, default: 1024<br/>(env: LLAMA_ARG_FIT_TARGET) |90| `-fitc, --fit-ctx N` | minimum ctx size that can be set by --fit option, default: 4096<br/>(env: LLAMA_ARG_FIT_CTX) |91| `--check-tensors` | check model tensor data for invalid values (default: false) |92| `--override-kv KEY=TYPE:VALUE,...` | advanced option to override model metadata by key. to specify multiple overrides, either use comma-separated values.<br/>types: int, float, bool, str. example: --override-kv tokenizer.ggml.add_bos_token=bool:false,tokenizer.ggml.add_eos_token=bool:false |93| `--op-offload, --no-op-offload` | whether to offload host tensor operations to device (default: true) |94| `--lora FNAME` | path to LoRA adapter (use comma-separated values to load multiple adapters) |95| `--lora-scaled FNAME:SCALE,...` | path to LoRA adapter with user defined scaling (format: FNAME:SCALE,...)<br/>note: use comma-separated values |96| `--control-vector FNAME` | add a control vector<br/>note: use comma-separated values to add multiple control vectors |97| `--control-vector-scaled FNAME:SCALE,...` | add a control vector with user defined scaling SCALE<br/>note: use comma-separated values (format: FNAME:SCALE,...) |98| `--control-vector-layer-range START END` | layer range to apply the control vector(s) to, start and end inclusive |99| `-m, --model FNAME` | model path to load<br/>(env: LLAMA_ARG_MODEL) |100| `-mu, --model-url MODEL_URL` | model download url (default: unused)<br/>(env: LLAMA_ARG_MODEL_URL) |101| `-dr, --docker-repo [<repo>/]<model>[:quant]` | Docker Hub model repository. repo is optional, default to ai/. quant is optional, default to :latest.<br/>example: gemma3<br/>(default: unused)<br/>(env: LLAMA_ARG_DOCKER_REPO) |102| `-hf, -hfr, --hf-repo <user>/<model>[:quant]` | Hugging Face model repository; quant is optional, case-insensitive, default to Q4_K_M, or falls back to the first file in the repo if Q4_K_M doesn't exist.<br/>mmproj is also downloaded automatically if available. to disable, add --no-mmproj<br/>example: ggml-org/GLM-4.7-Flash-GGUF:Q4_K_M<br/>(default: unused)<br/>(env: LLAMA_ARG_HF_REPO) |103| `-hff, --hf-file FILE` | Hugging Face model file. If specified, it will override the quant in --hf-repo (default: unused)<br/>(env: LLAMA_ARG_HF_FILE) |104| `-hft, --hf-token TOKEN` | Hugging Face access token (default: value from HF_TOKEN environment variable)<br/>(env: HF_TOKEN) |105| `--log-disable` | Log disable |106| `--log-file FNAME` | Log to file<br/>(env: LLAMA_ARG_LOG_FILE) |107| `--log-jsonl, --no-log-jsonl` | Log as JSONL (one JSON object per line) to stdout, this also disables colored logging (default: disabled)<br/>(env: LLAMA_ARG_LOG_JSONL) |108| `--log-colors [on\|off\|auto]` | Set colored logging ('on', 'off', or 'auto', default: 'auto')<br/>'auto' enables colors when output is to a terminal<br/>(env: LLAMA_ARG_LOG_COLORS) |109| `-v, --verbose, --log-verbose` | Set verbosity level to infinity (i.e. log all messages, useful for debugging) |110| `--offline` | Offline mode: forces use of cache, prevents network access<br/>(env: LLAMA_ARG_OFFLINE) |111| `-lv, --verbosity, --log-verbosity N` | Set the verbosity threshold. Messages with a higher verbosity will be ignored. Values:<br/> - 0: generic output<br/> - 1: error<br/> - 2: warning<br/> - 3: info<br/> - 4: trace (more info)<br/> - 5: debug<br/>(default: 3)<br/><br/>(env: LLAMA_ARG_LOG_VERBOSITY) |112| `--log-prefix, --no-log-prefix` | Enable prefix in log messages<br/>(env: LLAMA_ARG_LOG_PREFIX) |113| `--log-timestamps, --no-log-timestamps` | Enable timestamps in log messages<br/>(env: LLAMA_ARG_LOG_TIMESTAMPS) |114| `--spec-draft-type-k, -ctkd, --cache-type-k-draft TYPE` | KV cache data type for K for the draft model<br/>allowed values: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1<br/>(default: f16)<br/>(env: LLAMA_ARG_SPEC_DRAFT_CACHE_TYPE_K) |115| `--spec-draft-type-v, -ctvd, --cache-type-v-draft TYPE` | KV cache data type for V for the draft model<br/>allowed values: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1<br/>(default: f16)<br/>(env: LLAMA_ARG_SPEC_DRAFT_CACHE_TYPE_V) |116 117 118### Sampling params119 120| Argument | Explanation |121| -------- | ----------- |122| `--samplers SAMPLERS` | samplers that will be used for generation in the order, separated by ';'<br/>(default: penalties;dry;top_n_sigma;top_k;typ_p;top_p;min_p;xtc;temperature) |123| `-s, --seed SEED` | RNG seed (default: -1, use random seed for -1) |124| `--sampler-seq, --sampling-seq SEQUENCE` | simplified sequence for samplers that will be used (default: edskypmxt) |125| `--ignore-eos` | ignore end of stream token and continue generating (implies --logit-bias EOS-inf) |126| `--temp, --temperature N` | temperature (default: 0.80) |127| `--top-k N` | top-k sampling (default: 40, 0 = disabled)<br/>(env: LLAMA_ARG_TOP_K) |128| `--top-p N` | top-p sampling (default: 0.95, 1.0 = disabled) |129| `--min-p N` | min-p sampling (default: 0.05, 0.0 = disabled) |130| `--top-nsigma, --top-n-sigma N` | top-n-sigma sampling (default: -1.00, -1.0 = disabled) |131| `--xtc-probability N` | xtc probability (default: 0.00, 0.0 = disabled) |132| `--xtc-threshold N` | xtc threshold (default: 0.10, 1.0 = disabled) |133| `--typical, --typical-p N` | locally typical sampling, parameter p (default: 1.00, 1.0 = disabled) |134| `--repeat-last-n N` | last n tokens to consider for penalize (default: 64, 0 = disabled) |135| `--repeat-penalty N` | penalize repeat sequence of tokens (default: 1.00, 1.0 = disabled) |136| `--presence-penalty N` | repeat alpha presence penalty (default: 0.00, 0.0 = disabled) |137| `--frequency-penalty N` | repeat alpha frequency penalty (default: 0.00, 0.0 = disabled) |138| `--dry-multiplier N` | set DRY sampling multiplier (default: 0.00, 0.0 = disabled) |139| `--dry-base N` | set DRY sampling base value (default: 1.75) |140| `--dry-allowed-length N` | set allowed length for DRY sampling (default: 2) |141| `--dry-penalty-last-n N` | set DRY penalty for the last n tokens (default: 64, 0 = disable) |142| `--dry-sequence-breaker STRING` | add sequence breaker for DRY sampling, clearing out default breakers ('\n', ':', '"', '*') in the process; use "none" to not use any sequence breakers |143| `--adaptive-target N` | adaptive-p: select tokens near this probability (valid range 0.0 to 1.0; negative = disabled) (default: -1.00)<br/>[(more info)](https://github.com/ggml-org/llama.cpp/pull/17927) |144| `--adaptive-decay N` | adaptive-p: decay rate for target adaptation over time. lower values are more reactive, higher values are more stable.<br/>(valid range 0.0 to 0.99) (default: 0.90) |145| `--dynatemp-range N` | dynamic temperature range (default: 0.00, 0.0 = disabled) |146| `--dynatemp-exp N` | dynamic temperature exponent (default: 1.00) |147| `--mirostat N` | use Mirostat sampling.<br/>Top K, Nucleus and Locally Typical samplers are ignored if used.<br/>(default: 0, 0 = disabled, 1 = Mirostat, 2 = Mirostat 2.0) |148| `--mirostat-lr N` | Mirostat learning rate, parameter eta (default: 0.10) |149| `--mirostat-ent N` | Mirostat target entropy, parameter tau (default: 5.00) |150| `-l, --logit-bias TOKEN_ID(+/-)BIAS` | modifies the likelihood of token appearing in the completion,<br/>i.e. `--logit-bias 15043+1` to increase likelihood of token ' Hello',<br/>or `--logit-bias 15043-1` to decrease likelihood of token ' Hello' |151| `--grammar GRAMMAR` | BNF-like grammar to constrain generations (see samples in grammars/ dir) |152| `--grammar-file FNAME` | file to read grammar from |153| `-j, --json-schema SCHEMA` | JSON schema to constrain generations (https://json-schema.org/), e.g. `{"type": "object"}` for any JSON object |154| `-jf, --json-schema-file FILE` | File containing a JSON schema to constrain generations (https://json-schema.org/), e.g. `{"type": "object"}` for any JSON object |155| `-bs, --backend-sampling` | enable backend sampling (experimental) (default: disabled)<br/>(env: LLAMA_ARG_BACKEND_SAMPLING) |156 157 158### Server-specific params159 160| Argument | Explanation |161| -------- | ----------- |162| `-lcs, --lookup-cache-static FNAME` | path to static lookup cache to use for lookup decoding (not updated by generation) |163| `-lcd, --lookup-cache-dynamic FNAME` | path to dynamic lookup cache to use for lookup decoding (updated by generation) |164| `--kv-unified-per-slot N` | context limit per parallel slot (default: unset, behavior unchanged).<br/>when set without -c/--ctx-size, the shared KV pool is sized to n_parallel*N<br/>(env: LLAMA_ARG_KV_UNIFIED_PER_SLOT) |165| `-ctxcp, --ctx-checkpoints, --swa-checkpoints N` | max number of context checkpoints to create per slot (default: 32)[(more info)](https://github.com/ggml-org/llama.cpp/pull/15293)<br/>(env: LLAMA_ARG_CTX_CHECKPOINTS) |166| `-cms, --checkpoint-min-step N` | minimum spacing between context checkpoints in tokens (default: 8192, 0 = no minimum)<br/>(env: LLAMA_ARG_CHECKPOINT_MIN_SPACING_NT) |167| `-cram, --cache-ram N` | set the maximum cache size in MiB (default: 8192, -1 - no limit, 0 - disable)[(more info)](https://github.com/ggml-org/llama.cpp/pull/16391)<br/>(env: LLAMA_ARG_CACHE_RAM) |168| `-kvu, --kv-unified, -no-kvu, --no-kv-unified` | use single unified KV buffer shared across all sequences (default: enabled if number of slots is auto)<br/>(env: LLAMA_ARG_KV_UNIFIED) |169| `--cache-idle-slots, --no-cache-idle-slots` | save idle slots to the prompt cache on new task, and clear them when using unified KV (default: enabled, requires cache-ram)<br/>(env: LLAMA_ARG_CACHE_IDLE_SLOTS) |170| `--context-shift, --no-context-shift` | whether to use context shift on infinite text generation (default: disabled)<br/>(env: LLAMA_ARG_CONTEXT_SHIFT) |171| `-r, --reverse-prompt PROMPT` | halt generation at PROMPT, return control in interactive mode |172| `-sp, --special` | special tokens output enabled (default: false) |173| `--warmup, --no-warmup` | whether to perform warmup with an empty run (default: enabled) |174| `--spm-infill` | use Suffix/Prefix/Middle pattern for infill (instead of Prefix/Suffix/Middle) as some models prefer this. (default: disabled) |175| `--pooling {none,mean,cls,last,rank}` | pooling type for embeddings, use model default if unspecified<br/>(env: LLAMA_ARG_POOLING) |176| `-np, --parallel N` | number of server slots (default: -1, -1 = auto)<br/>(env: LLAMA_ARG_N_PARALLEL) |177| `-cb, --cont-batching, -nocb, --no-cont-batching` | whether to enable continuous batching (a.k.a dynamic batching) (default: enabled)<br/>(env: LLAMA_ARG_CONT_BATCHING) |178| `-mm, --mmproj FILE` | path to a multimodal projector file. see tools/mtmd/README.md<br/>note: if -hf is used, this argument can be omitted<br/>(env: LLAMA_ARG_MMPROJ) |179| `-mmu, --mmproj-url URL` | URL to a multimodal projector file. see tools/mtmd/README.md<br/>(env: LLAMA_ARG_MMPROJ_URL) |180| `--mmproj-auto, --no-mmproj, --no-mmproj-auto` | whether to use multimodal projector file (if available), useful when using -hf (default: enabled)<br/>(env: LLAMA_ARG_MMPROJ_AUTO) |181| `--mmproj-offload, --no-mmproj-offload` | whether to enable GPU offloading for multimodal projector (default: enabled)<br/>(env: LLAMA_ARG_MMPROJ_OFFLOAD) |182| `-mmdev, --mmproj-device DEVICE` | device to use for multimodal projector (none = don't offload, default: follows --device)<br/>use --list-devices to see a list of available devices<br/>(env: MTMD_BACKEND_DEVICE) |183| `--image-min-tokens N` | minimum number of tokens each image can take, only used by vision models with dynamic resolution (default: read from model)<br/>(env: LLAMA_ARG_IMAGE_MIN_TOKENS) |184| `--image-max-tokens N` | maximum number of tokens each image can take, only used by vision models with dynamic resolution (default: read from model)<br/>(env: LLAMA_ARG_IMAGE_MAX_TOKENS) |185| `--mtmd-batch-max-tokens N` | maximum number of image tokens per batch when encoding images (default: 1024)<br/>(env: LLAMA_ARG_MTMD_BATCH_MAX_TOKENS) |186| `--video-fps N` | target video frame rate (default: 4.0)<br/>(env: LLAMA_ARG_VIDEO_FPS) |187| `--video-timestamp-interval N` | interval in milliseconds between text timestamps (default: 5000)<br/>(env: LLAMA_ARG_VIDEO_TIMESTAMP_INTERVAL) |188| `--video-ffmpeg-dir DIR` | path to the directory containing ffmpeg and ffprobe (default: search in PATH)<br/>(env: LLAMA_ARG_VIDEO_FFMPEG_DIR) |189| `-a, --alias STRING` | set model name aliases, comma-separated (to be used by API)<br/>(env: LLAMA_ARG_ALIAS) |190| `--tags STRING` | set model tags, comma-separated (informational, not used for routing)<br/>(env: LLAMA_ARG_TAGS) |191| `--embd-normalize N` | normalisation for embeddings (default: 2) (-1=none, 0=max absolute int16, 1=taxicab, 2=euclidean, >2=p-norm) |192| `--host HOST` | ip address to listen, or bind to an UNIX socket if the address ends with .sock (default: 127.0.0.1)<br/>(env: LLAMA_ARG_HOST) |193| `--port PORT` | port to listen (default: 8080)<br/>(env: LLAMA_ARG_PORT) |194| `--reuse-port` | allow multiple sockets to bind to the same port (default: disabled)<br/>(env: LLAMA_ARG_REUSE_PORT) |195| `--path PATH` | path to serve static files from (default: )<br/>(env: LLAMA_ARG_STATIC_PATH) |196| `--cors-origins ORIGINS` | comma-separated list of allowed origins for CORS (default: *)<br/>if set to special value 'localhost', reflect the Origin header only if it is localhost<br/>(env: LLAMA_ARG_CORS_ORIGINS) |197| `--cors-methods METHODS` | comma-separated list of allowed methods for CORS (default: GET, POST, DELETE, OPTIONS)<br/>(env: LLAMA_ARG_CORS_METHODS) |198| `--cors-headers HEADERS` | comma-separated list of allowed headers for CORS (default: *)<br/>(env: LLAMA_ARG_CORS_HEADERS) |199| `--cors-credentials, --no-cors-credentials` | whether to allow credentials for CORS (default: enabled)<br/>note: if this is enabled and --cors-origins is set to * (default), the Origin header will be echoed back, and credentials will always be allowed<br/>(env: LLAMA_ARG_CORS_CREDENTIALS) |200| `--api-prefix PREFIX` | prefix path the server serves from, without the trailing slash (default: )<br/>(env: LLAMA_ARG_API_PREFIX) |201| `--ui-config, --webui-config JSON` | JSON that provides default UI settings (overrides UI defaults)<br/>(env: LLAMA_ARG_UI_CONFIG) |202| `--ui-config-file, --webui-config-file PATH` | JSON file that provides default UI settings (overrides UI defaults)<br/>(env: LLAMA_ARG_UI_CONFIG_FILE) |203| `--ui-mcp-proxy, --webui-mcp-proxy, --no-ui-mcp-proxy, --no-webui-mcp-proxy` | experimental: whether to enable MCP CORS proxy - do not enable in untrusted environments (default: disabled)<br/>(env: LLAMA_ARG_UI_MCP_PROXY) |204| `--tools TOOL1,TOOL2,...` | experimental: whether to enable built-in tools for AI agents - do not enable in untrusted environments (default: no tools)<br/>specify "all" to enable all tools<br/>available tools: read_file, file_glob_search, grep_search, exec_shell_command, write_file, edit_file, get_info<br/>note: for security reasons, this will limit --cors-origins to localhost by default<br/>(env: LLAMA_ARG_TOOLS) |205| `--tools-runtime OPTION` | experimental: run tools in a separate runtime environment (default: none, use host environment)<br/>available options:<br/>  'docker:<image>', 'podman:<image>': spin up a new container and reuse it for all invocations, clean up on server exit<br/>  'docker-container:<id>', 'podman-container:<id>': use an existing container by ID, won't stop on server exit<br/>  'ssh:<target>': run tools on a remote POSIX host over SSH, key-based auth and a trusted host key are required<br/><br/>(env: LLAMA_ARG_TOOLS_RUNTIME) |206| `--mcp-servers-config PATH` | experimental: path to JSON file with MCP server definitions (Cursor-compatible format) - do not enable in untrusted environments (default: none)<br/>note: for security reasons, this will limit --cors-origins to localhost by default<br/>(env: LLAMA_ARG_MCP_SERVERS_CONFIG) |207| `--mcp-servers-json JSON` | experimental: inline JSON with MCP server definitions (Cursor-compatible format) - do not enable in untrusted environments (default: none)<br/>note: for security reasons, this will limit --cors-origins to localhost by default<br/>(env: LLAMA_ARG_MCP_SERVERS_JSON) |208| `-ag, --agent, -no-ag, --no-agent` | whether to enable CORS proxy and all built-in tools - do not enable in untrusted environments (default: disabled)<br/>note: for security reasons, this will limit --cors-origins to localhost by default<br/>(env: LLAMA_ARG_AGENT) |209| `--ui, --webui, --no-ui, --no-webui` | whether to enable the Web UI (default: enabled)<br/>(env: LLAMA_ARG_UI) |210| `--embedding, --embeddings` | restrict to only support embedding use case; use only with dedicated embedding models (default: disabled)<br/>(env: LLAMA_ARG_EMBEDDINGS) |211| `--rerank, --reranking` | enable reranking endpoint on server (default: disabled)<br/>(env: LLAMA_ARG_RERANKING) |212| `--api-key KEY` | API key to use for authentication, multiple keys can be provided as a comma-separated list (default: none)<br/>(env: LLAMA_API_KEY) |213| `--api-key-file FNAME` | path to file containing API keys, one per line; lines starting with a hash are treated as comments (default: none)<br/>(env: LLAMA_ARG_API_KEY_FILE) |214| `--ssl-key-file FNAME` | path to file a PEM-encoded SSL private key<br/>(env: LLAMA_ARG_SSL_KEY_FILE) |215| `--ssl-cert-file FNAME` | path to file a PEM-encoded SSL certificate<br/>(env: LLAMA_ARG_SSL_CERT_FILE) |216| `--chat-template-kwargs STRING` | sets additional params for the json template parser, must be a valid json object string, e.g. '{"key1":"value1","key2":"value2"}'<br/>(env: LLAMA_ARG_CHAT_TEMPLATE_KWARGS) |217| `-to, --timeout N` | server read/write timeout in seconds (default: 3600)<br/>(env: LLAMA_ARG_TIMEOUT) |218| `--sse-ping-interval N` | server SSE ping interval in seconds (-1 = disabled, default: 30)<br/>(env: LLAMA_ARG_SSE_PING_INTERVAL) |219| `--threads-http N` | number of threads used to process HTTP requests (default: -1)<br/>(env: LLAMA_ARG_THREADS_HTTP) |220| `--cache-prompt, --no-cache-prompt` | whether to enable prompt caching (default: enabled)<br/>(env: LLAMA_ARG_CACHE_PROMPT) |221| `--cache-reuse N` | min chunk size to attempt reusing from the cache via KV shifting, requires prompt caching to be enabled (default: 0)<br/>[(card)](https://ggml.ai/f0.png)<br/>(env: LLAMA_ARG_CACHE_REUSE) |222| `--metrics` | enable prometheus compatible metrics endpoint (default: disabled)<br/>(env: LLAMA_ARG_ENDPOINT_METRICS) |223| `--props` | enable changing global properties via POST /props (default: disabled)<br/>(env: LLAMA_ARG_ENDPOINT_PROPS) |224| `--slots, --no-slots` | expose slots monitoring endpoint (default: enabled)<br/>(env: LLAMA_ARG_ENDPOINT_SLOTS) |225| `--slot-save-path PATH` | path to save slot kv cache (default: disabled) |226| `--media-path PATH` | directory for loading local media files; files can be accessed via file:// URLs using relative paths (default: disabled) |227| `--models-dir PATH` | directory containing models for the router server (default: disabled)<br/>(env: LLAMA_ARG_MODELS_DIR) |228| `--models-preset PATH` | path to INI file containing model presets for the router server (default: disabled)<br/>(env: LLAMA_ARG_MODELS_PRESET) |229| `--models-max N` | for router server, maximum number of models to load simultaneously (default: 4, 0 = unlimited)<br/>(env: LLAMA_ARG_MODELS_MAX) |230| `--models-autoload, --no-models-autoload` | for router server, whether to automatically load models (default: enabled)<br/>(env: LLAMA_ARG_MODELS_AUTOLOAD) |231| `--jinja, --no-jinja` | whether to use jinja template engine for chat (default: enabled)<br/>(env: LLAMA_ARG_JINJA) |232| `--reasoning-format FORMAT` | controls whether thought tags are allowed and/or extracted from the response, and in which format they're returned; one of:<br/>- none: leaves thoughts unparsed in `message.content`<br/>- deepseek: puts thoughts in `message.reasoning_content`<br/>- deepseek-legacy: keeps `<think>` tags in `message.content` while also populating `message.reasoning_content`<br/>(default: auto)<br/>(env: LLAMA_ARG_THINK) |233| `-rea, --reasoning [on\|off\|auto]` | Use reasoning/thinking in the chat ('on', 'off', or 'auto', default: 'auto' (detect from template))<br/>(env: LLAMA_ARG_REASONING) |234| `--reasoning-effort LEVEL` | reasoning effort level given to the chat template: 'default' to keep the template default,<br/>or a level such as 'minimal', 'low', 'medium', 'high', 'xhigh' or 'max' (default: default)<br/>(env: LLAMA_ARG_REASONING_EFFORT) |235| `--reasoning-budget N` | token budget for thinking: -1 for unrestricted, 0 for immediate end, N>0 for token budget (default: -1)<br/>(env: LLAMA_ARG_THINK_BUDGET) |236| `--reasoning-budget-message MESSAGE` | message injected before the end-of-thinking tag when reasoning budget is exhausted (default: none)<br/>(env: LLAMA_ARG_THINK_BUDGET_MESSAGE) |237| `--reasoning-preserve, --no-reasoning-preserve` | preserve reasoning trace in the full history, not just the last assistant message (default: enabled)<br/>compatible with certain templates having 'supports_preserve_reasoning' capability<br/>example: https://docs.z.ai/guides/capabilities/thinking-mode#preserved-thinking<br/>(env: LLAMA_ARG_REASONING_PRESERVE) |238| `--chat-template JINJA_TEMPLATE` | set custom jinja chat template (default: template taken from model's metadata)<br/>if suffix/prefix are specified, template will be disabled<br/>only commonly used templates are accepted (unless --jinja is set before this flag):<br/>list of built-in templates:<br/>bailing, bailing-think, bailing2, chatglm3, chatglm4, chatml, command-r, deepseek, deepseek-ocr, deepseek2, deepseek3, exaone-moe, exaone3, exaone4, falcon3, gemma, gigachat, glmedge, gpt-oss, granite, granite-4.0, granite-4.1, grok-2, hunyuan-dense, hunyuan-moe, hunyuan-vl, kimi-k2, llama2, llama2-sys, llama2-sys-bos, llama2-sys-strip, llama3, llama4, megrez, minicpm, mistral-v1, mistral-v3, mistral-v3-tekken, mistral-v7, mistral-v7-tekken, monarch, openchat, orion, pangu-embedded, phi3, phi4, rwkv-world, seed_oss, smolvlm, solar-open, vicuna, vicuna-orca, yandex, zephyr<br/>(env: LLAMA_ARG_CHAT_TEMPLATE) |239| `--chat-template-file JINJA_TEMPLATE_FILE` | set custom jinja chat template file (default: template taken from model's metadata)<br/>if suffix/prefix are specified, template will be disabled<br/>only commonly used templates are accepted (unless --jinja is set before this flag):<br/>list of built-in templates:<br/>bailing, bailing-think, bailing2, chatglm3, chatglm4, chatml, command-r, deepseek, deepseek-ocr, deepseek2, deepseek3, exaone-moe, exaone3, exaone4, falcon3, gemma, gigachat, glmedge, gpt-oss, granite, granite-4.0, granite-4.1, grok-2, hunyuan-dense, hunyuan-moe, hunyuan-vl, kimi-k2, llama2, llama2-sys, llama2-sys-bos, llama2-sys-strip, llama3, llama4, megrez, minicpm, mistral-v1, mistral-v3, mistral-v3-tekken, mistral-v7, mistral-v7-tekken, monarch, openchat, orion, pangu-embedded, phi3, phi4, rwkv-world, seed_oss, smolvlm, solar-open, vicuna, vicuna-orca, yandex, zephyr<br/>(env: LLAMA_ARG_CHAT_TEMPLATE_FILE) |240| `--skip-chat-parsing, --no-skip-chat-parsing` | force a pure content parser, even if a Jinja template is specified; model will output everything in the content section, including any reasoning and/or tool calls (default: disabled)<br/>(env: LLAMA_ARG_SKIP_CHAT_PARSING) |241| `--prefill-assistant, --no-prefill-assistant` | whether to prefill the assistant's response if the last message is an assistant message (default: prefill enabled)<br/>when this flag is set, if the last message is an assistant message then it will be treated as a full message and not prefilled<br/><br/>(env: LLAMA_ARG_PREFILL_ASSISTANT) |242| `-sps, --slot-prompt-similarity SIMILARITY` | how much the prompt of a request must match the prompt of a slot in order to use that slot (default: 0.10, 0.0 = disabled) |243| `--lora-init-without-apply` | load LoRA adapters without applying them (apply later via POST /lora-adapters) (default: disabled) |244| `--sleep-idle-seconds SECONDS` | number of seconds of idleness after which the server will sleep (default: -1; -1 = disabled) |245| `--log-prompts-dir PATH` | Log prompts to directory (auto-created if not present; only used for debugging, default: disabled) |246| `--spec-draft-hf, -hfd, -hfrd, --hf-repo-draft <user>/<model>[:quant]` | Same as --hf-repo, but for the draft model (default: unused)<br/>(env: LLAMA_ARG_SPEC_DRAFT_HF_REPO) |247| `--spec-draft-threads, -td, --threads-draft N` | number of threads to use during generation (default: same as --threads) |248| `--spec-draft-threads-batch, -tbd, --threads-batch-draft N` | number of threads to use during batch and prompt processing (default: same as --threads-draft) |249| `--spec-draft-cpu-mask, -Cd, --cpu-mask-draft M` | Draft model CPU affinity mask. Complements cpu-range-draft (default: same as --cpu-mask) |250| `--spec-draft-cpu-range, -Crd, --cpu-range-draft lo-hi` | Ranges of CPUs for affinity. Complements --cpu-mask-draft |251| `--spec-draft-cpu-strict, --cpu-strict-draft <0\|1>` | Use strict CPU placement for draft model (default: same as --cpu-strict) |252| `--spec-draft-prio, --prio-draft N` | set draft process/thread priority : 0-normal, 1-medium, 2-high, 3-realtime (default: 0) |253| `--spec-draft-poll, --poll-draft <0\|1>` | Use polling to wait for draft model work (default: same as --poll) |254| `--spec-draft-cpu-mask-batch, -Cbd, --cpu-mask-batch-draft M` | Draft model CPU affinity mask. Complements cpu-range-draft (default: same as --cpu-mask) |255| `--spec-draft-cpu-strict-batch, --cpu-strict-batch-draft <0\|1>` | Use strict CPU placement for draft model (default: --cpu-strict-draft) |256| `--spec-draft-prio-batch, --prio-batch-draft N` | set draft process/thread priority : 0-normal, 1-medium, 2-high, 3-realtime (default: 0) |257| `--spec-draft-poll-batch, --poll-batch-draft <0\|1>` | Use polling to wait for draft model work (default: --poll-draft) |258| `--spec-draft-override-tensor, -otd, --override-tensor-draft <tensor name pattern>=<buffer type>,...` | override tensor buffer type for draft model |259| `--spec-draft-cpu-moe, -cmoed, --cpu-moe-draft` | keep all Mixture of Experts (MoE) weights in the CPU for the draft model<br/>(env: LLAMA_ARG_SPEC_DRAFT_CPU_MOE) |260| `--spec-draft-n-cpu-moe, --spec-draft-ncmoe, -ncmoed, --n-cpu-moe-draft N` | keep the Mixture of Experts (MoE) weights of the first N layers in the CPU for the draft model<br/>(env: LLAMA_ARG_SPEC_DRAFT_N_CPU_MOE) |261| `--spec-draft-n-max N` | number of tokens to draft for speculative decoding (default: 3)<br/>(env: LLAMA_ARG_SPEC_DRAFT_N_MAX) |262| `--spec-draft-n-min N` | minimum number of draft tokens to use for speculative decoding (default: 0)<br/>(env: LLAMA_ARG_SPEC_DRAFT_N_MIN) |263| `--spec-synth-len L` | target mean synthetic acceptance length, including the target token (benchmarking only)<br/>(env: LLAMA_ARG_SPEC_SYNTH_LEN) |264| `--spec-synth-rates P0,P1,...` | comma-separated unconditional per-position synthetic acceptance probabilities (benchmarking only)<br/>(env: LLAMA_ARG_SPEC_SYNTH_RATES) |265| `--spec-draft-p-split, --draft-p-split P` | speculative decoding split probability (default: 0.10)<br/>(env: LLAMA_ARG_SPEC_DRAFT_P_SPLIT) |266| `--spec-draft-p-min, --draft-p-min P` | minimum speculative decoding probability (greedy) (default: 0.00)<br/>(env: LLAMA_ARG_SPEC_DRAFT_P_MIN) |267| `--spec-draft-backend-sampling, --no-spec-draft-backend-sampling` | offload draft sampling to the backend (default: enabled)<br/>(env: LLAMA_ARG_SPEC_DRAFT_BACKEND_SAMPLING) |268| `--spec-draft-device, -devd, --device-draft <dev1,dev2,..>` | comma-separated list of devices to use for offloading the draft model (none = don't offload, default: follows --device)<br/>use --list-devices to see a list of available devices |269| `--spec-draft-ngl, -ngld, --gpu-layers-draft, --n-gpu-layers-draft N` | max. number of draft model layers to store in VRAM, either an exact number, 'auto', or 'all' (default: auto)<br/>(env: LLAMA_ARG_N_GPU_LAYERS_DRAFT) |270| `--spec-draft-model, -md, --model-draft FNAME` | draft model for speculative decoding (default: unused)<br/>(env: LLAMA_ARG_SPEC_DRAFT_MODEL) |271| `--spec-type none,draft-simple,draft-eagle3,draft-mtp,draft-dflash,draft-dspark,ngram-simple,ngram-map-k,ngram-map-k4v,ngram-mod,ngram-cache` | comma-separated list of types of speculative decoding to use (default: none)<br/><br/>(env: LLAMA_ARG_SPEC_TYPE) |272| `--spec-ngram-mod-n-min N` | minimum number of ngram tokens to use for ngram-based speculative decoding (default: 48) |273| `--spec-ngram-mod-n-max N` | maximum number of ngram tokens to use for ngram-based speculative decoding (default: 64) |274| `--spec-ngram-mod-n-match N` | ngram-mod lookup length (default: 24) |275| `--spec-ngram-simple-size-n N` | ngram size N for ngram-simple speculative decoding, length of lookup n-gram (default: 12) |276| `--spec-ngram-simple-size-m N` | ngram size M for ngram-simple speculative decoding, length of draft m-gram (default: 48) |277| `--spec-ngram-simple-min-hits N` | minimum hits for ngram-simple speculative decoding (default: 1) |278| `--spec-ngram-map-k-size-n N` | ngram size N for ngram-map-k speculative decoding, length of lookup n-gram (default: 12) |279| `--spec-ngram-map-k-size-m N` | ngram size M for ngram-map-k speculative decoding, length of draft m-gram (default: 48) |280| `--spec-ngram-map-k-min-hits N` | minimum hits for ngram-map-k speculative decoding (default: 1) |281| `--spec-ngram-map-k4v-size-n N` | ngram size N for ngram-map-k4v speculative decoding, length of lookup n-gram (default: 12) |282| `--spec-ngram-map-k4v-size-m N` | ngram size M for ngram-map-k4v speculative decoding, length of draft m-gram (default: 48) |283| `--spec-ngram-map-k4v-min-hits N` | minimum hits for ngram-map-k4v speculative decoding (default: 1) |284| `--draft, --draft-n, --draft-max N` | the argument has been removed. use --spec-draft-n-max or --spec-ngram-mod-n-max<br/>(env: LLAMA_ARG_DRAFT_MAX) |285| `--draft-min, --draft-n-min N` | the argument has been removed. use --spec-draft-n-min or --spec-ngram-mod-n-min<br/>(env: LLAMA_ARG_DRAFT_MIN) |286| `--spec-ngram-size-n N` | the argument has been removed. use the respective --spec-ngram-*-size-n or --spec-ngram-mod-n-match |287| `--spec-ngram-size-m N` | the argument has been removed. use the respective --spec-ngram-*-size-m |288| `--spec-ngram-min-hits N` | the argument has been removed. use the respective --spec-ngram-*-min-hits |289| `--embd-gemma-default` | use default EmbeddingGemma model (note: can download weights from the internet) |290| `--fim-qwen-1.5b-default` | use default Qwen 2.5 Coder 1.5B (note: can download weights from the internet) |291| `--fim-qwen-3b-default` | use default Qwen 2.5 Coder 3B (note: can download weights from the internet) |292| `--fim-qwen-7b-default` | use default Qwen 2.5 Coder 7B (note: can download weights from the internet) |293| `--fim-qwen-7b-spec` | use Qwen 2.5 Coder 7B + 0.5B draft for speculative decoding (note: can download weights from the internet) |294| `--fim-qwen-14b-spec` | use Qwen 2.5 Coder 14B + 0.5B draft for speculative decoding (note: can download weights from the internet) |295| `--fim-qwen-30b-default` | use default Qwen 3 Coder 30B A3B Instruct (note: can download weights from the internet) |296| `--gpt-oss-20b-default` | use gpt-oss-20b (note: can download weights from the internet) |297| `--gpt-oss-120b-default` | use gpt-oss-120b (note: can download weights from the internet) |298| `--vision-gemma-4b-default` | use Gemma 3 4B QAT (note: can download weights from the internet) |299| `--vision-gemma-12b-default` | use Gemma 3 12B QAT (note: can download weights from the internet) |300| `--spec-default` | enable default speculative decoding config |301 302<!-- HELP_END -->303 304Note: If both command line argument and environment variable are both set for the same param, the argument will take precedence over env var.305 306For string options like `--load-mode`, the environment variable is handled as shown in this example:307- `LLAMA_ARG_LOAD_MODE=auto` sets the loading mode to auto (default)308- `LLAMA_ARG_LOAD_MODE=none` disables special loading309- `LLAMA_ARG_LOAD_MODE=mmap` enables memory-mapping310- `LLAMA_ARG_LOAD_MODE=mlock` locks the model in RAM311- `LLAMA_ARG_LOAD_MODE=mmap+mlock` enables memory-mapping and locks in RAM312- `LLAMA_ARG_LOAD_MODE=dio` uses DirectIO if available313 314For boolean options like `--kv-offload`:315- `LLAMA_ARG_KV_OFFLOAD=true` means enabled, other accepted values are: `1`, `on`, `enabled`316- `LLAMA_ARG_KV_OFFLOAD=false` means disabled, other accepted values are: `0`, `off`, `disabled`317 318Example usage of docker compose with environment variables:319 320```yml321services:322  llamacpp-server:323    image: ghcr.io/ggml-org/llama.cpp:server324    ports:325      - 8080:8080326    volumes:327      - ./models:/models328    environment:329      # alternatively, you can use "LLAMA_ARG_MODEL_URL" to download the model330      LLAMA_ARG_MODEL: /models/my_model.gguf331      LLAMA_ARG_CTX_SIZE: 4096332      LLAMA_ARG_N_PARALLEL: 2333      LLAMA_ARG_ENDPOINT_METRICS: 1334      LLAMA_ARG_PORT: 8080335```336 337### Multimodal support338 339Multimodal support was added in [#12898](https://github.com/ggml-org/llama.cpp/pull/12898) and is currently an experimental feature.340It is currently available in the following endpoints:341- The OAI-compatible chat endpoint.342- The non-OAI-compatible completions endpoint.343- The non-OAI-compatible embeddings endpoint.344 345For more details, please refer to [multimodal documentation](../../docs/multimodal.md)346 347### Server tools support348 349The server includes a set of server tools that enable the LLM to access the local file system directly from the Web UI.350 351To use this feature, start the server with `--tools all`. You can also enable only specific tools by passing a comma-separated list: `--tools name1,name2,...`. Run `--help` for the full list of available tool names.352 353### MCP servers354 355Besides the built-in tools, the server can expose tools coming from MCP servers, added in [#26062](https://github.com/ggml-org/llama.cpp/pull/26062). Only the stdio transport is supported: such a server is a child process reading JSON-RPC messages on its stdin and writing replies on its stdout, so nothing has to be started or maintained outside `llama-server`.356 357Servers are declared in a Cursor-compatible JSON file:358 359```json360{361  "mcpServers": {362    "example": { "command": "/path/to/server", "args": [] }363  }364}365```366 367```sh368llama-server -m model.gguf --mcp-servers-config mcp.json369```370 371The same JSON can be passed inline with `--mcp-servers-json`. Each entry under `mcpServers` accepts:372 373| Key | Explanation |374| --- | ----------- |375| `command` | executable to spawn, required, entries without it are skipped |376| `args` | array of arguments |377| `env` | object merged over the parent environment |378| `cwd` | working directory of the child process |379| `timeout_ms` | per-tool-call timeout (default: 30000) |380 381Every server is spawned once at startup to list its tools, then stopped, and respawned on demand when one of its tools is called. Tools are exposed as `<server>_<tool>` alongside the built-in ones: they show up in the Web UI and in `GET /tools`, and the model calls them like any other tool. A name colliding with an already registered tool is skipped. This is independent of `--tools`, MCP servers can be the only tools available.382 383The child process runs with the same privileges as the server, so only declare commands you trust. As with `--tools`, `--cors-origins` then defaults to `localhost`.384 385Note: `--ui-mcp-proxy` is unrelated, it only lets the Web UI reach remote MCP servers from the browser.386 387Any server written against the [MCP specification](https://modelcontextprotocol.io) works as is, whether it uses an official SDK or not: the transport is one JSON-RPC message per line on stdio, so a script wrapping an existing program is a valid server too.388 389### CORS390 391By default the server reflects any `Origin` header back with credentials allowed. This matches the old, always-on `*` behavior and is fine as long as the server only exposes stateless, read-only endpoints.392 393Enabling `--tools` or `--agent` exposes file read/write over the API, so in that case `--cors-origins` defaults to `localhost` instead: only pages served from localhost can reach the server. Pass `--cors-origins` explicitly to override either default.394 395Recommended `--cors-origins` setting, depending on where the server runs:396 397| Deployment | Recommendation |398| ---------- | --------------- |399| Public | set an API key, put the server behind a reverse proxy, `--cors-origins` optional |400| Local network | set `--cors-origins` to your frontend's origin |401| Same machine | `--cors-origins localhost` (default once `--agent` is set) |402 403Related flags: `--cors-origins`, `--cors-methods`, `--cors-headers`, `--cors-credentials` / `--no-cors-credentials`. Background and rationale: [#25655](https://github.com/ggml-org/llama.cpp/pull/25655).404 405## Build406 407`llama-server` is built alongside everything else from the root of the project408 409- Using `CMake`:410 411  ```bash412  cmake -B build413  cmake --build build --config Release -t llama-server414  ```415 416  Binary is at `./build/bin/llama-server`417 418## Build with SSL419 420`llama-server` can also be built with SSL support using OpenSSL 3421 422- Using `CMake`:423 424  ```bash425  cmake -B build -DLLAMA_OPENSSL=ON426  cmake --build build --config Release -t llama-server427  ```428 429## Quick Start430 431To get started right away, run the following command, making sure to use the correct path for the model you have:432 433### Unix-based systems (Linux, macOS, etc.)434 435```bash436./llama-server -m models/7B/ggml-model.gguf -c 2048437```438 439### Windows440 441```powershell442llama-server.exe -m models\7B\ggml-model.gguf -c 2048443```444 445The above command will start a server that by default listens on `127.0.0.1:8080`.446You can consume the endpoints with Postman or NodeJS with axios library. You can visit the web front end at the same url.447 448### Docker449 450```bash451docker run -p 8080:8080 -v /path/to/models:/models ghcr.io/ggml-org/llama.cpp:server -m models/7B/ggml-model.gguf -c 512 --host 0.0.0.0 --port 8080452 453# or, with CUDA:454docker run -p 8080:8080 -v /path/to/models:/models --gpus all ghcr.io/ggml-org/llama.cpp:server-cuda -m models/7B/ggml-model.gguf -c 512 --host 0.0.0.0 --port 8080 --n-gpu-layers 99455```456 457## Using with CURL458 459Using [curl](https://curl.se/). On Windows, `curl.exe` should be available in the base OS.460 461```sh462curl --request POST \463    --url http://localhost:8080/completion \464    --header "Content-Type: application/json" \465    --data '{"prompt": "Building a website can be done in 10 simple steps:","n_predict": 128}'466```467 468## API Endpoints469 470### GET `/health`: Returns health check result471 472This endpoint is public (no API key check). `/v1/health` also works.473 474**Response format**475 476- HTTP status code 503477  - Body: `{"error": {"code": 503, "message": "Loading model", "type": "unavailable_error"}}`478  - Explanation: the model is still being loaded.479- HTTP status code 200480  - Body: `{"status": "ok" }`481  - Explanation: the model is successfully loaded and the server is ready.482 483### POST `/completion`: Given a `prompt`, it returns the predicted completion.484 485> [!IMPORTANT]486>487> This endpoint is **not** OAI-compatible. For OAI-compatible client, use `/v1/completions` instead.488 489*Options:*490 491`prompt`: Provide the prompt for this completion as a string or as an array of strings or numbers representing tokens. Internally, if `cache_prompt` is `true`, the prompt is compared to the previous completion and only the "unseen" suffix is evaluated. A `BOS` token is inserted at the start, if all of the following conditions are true:492 493  - The prompt is a string or an array with the first element given as a string494  - The model's `tokenizer.ggml.add_bos_token` metadata is `true`495 496These input shapes and data type are allowed for `prompt`:497 498  - Single string: `"string"`499  - Single sequence of tokens: `[12, 34, 56]`500  - Mixed tokens and strings: `[12, 34, "string", 56, 78]`501  - A JSON object which optionally contains multimodal data: `{ "prompt_string": "string", "multimodal_data": ["base64"] }`502 503Multiple prompts are also supported. In this case, the completion result will be an array.504 505  - Only strings: `["string1", "string2"]`506  - Strings, JSON objects, and sequences of tokens: `["string1", [12, 34, 56], { "prompt_string": "string", "multimodal_data": ["base64"]}]`507  - Mixed types: `[[12, 34, "string", 56, 78], [12, 34, 56], "string", { "prompt_string": "string" }]`508 509Note for `multimodal_data` in JSON object prompts. This should be an array of strings, containing base64 encoded multimodal data such as images and audio. There must be an identical number of MTMD media markers in the string prompt element which act as placeholders for the data provided to this parameter. The multimodal data files will be substituted in order. The marker string (e.g. `<__media__>`) can be found by calling `mtmd_default_marker()` defined in [the MTMD C API](https://github.com/ggml-org/llama.cpp/blob/5fd160bbd9d70b94b5b11b0001fd7f477005e4a0/tools/mtmd/mtmd.h#L87). A client *must not* specify this field unless the server has the multimodal capability. Clients should check `/models` or `/v1/models` for the `multimodal` capability before a multimodal request.510 511`temperature`: Adjust the randomness of the generated text. Default: `0.8`512 513`dynatemp_range`: Dynamic temperature range. The final temperature will be in the range of `[temperature - dynatemp_range; temperature + dynatemp_range]` Default: `0.0`, which is disabled.514 515`dynatemp_exponent`: Dynamic temperature exponent. Default: `1.0`516 517`top_k`: Limit the next token selection to the K most probable tokens.  Default: `40`518 519`top_p`: Limit the next token selection to a subset of tokens with a cumulative probability above a threshold P. Default: `0.95`520 521`min_p`: The minimum probability for a token to be considered, relative to the probability of the most likely token. Default: `0.05`522 523`n_predict`: Set the maximum number of tokens to predict when generating text. **Note:** May exceed the set limit slightly if the last token is a partial multibyte character. When 0, no tokens will be generated but the prompt is evaluated into the cache. Default: `-1`, where `-1` is infinity.524 525`n_indent`: Specify the minimum line indentation for the generated text in number of whitespace characters. Useful for code completion tasks. Default: `0`526 527`n_keep`: Specify the number of tokens from the prompt to retain when the context size is exceeded and tokens need to be discarded. The number excludes the BOS token.528By default, this value is set to `0`, meaning no tokens are kept. Use `-1` to retain all tokens from the prompt.529 530`n_cmpl`: Number of completions to generate from the current prompt. If input has multiple prompts, the output will have N prompts times `n_cmpl` entries.531 532`n_cache_reuse`: Min chunk size to attempt reusing from the cache via KV shifting. For more info, see `--cache-reuse` arg. Default: `0`, which is disabled.533 534`stream`: Allows receiving each predicted token in real-time instead of waiting for the completion to finish (uses a different response format). To enable this, set to `true`.535 536`stop`: Specify a JSON array of stopping strings.537These words will not be included in the completion, so make sure to add them to the prompt for the next iteration. Default: `[]`538 539`typical_p`: Enable locally typical sampling with parameter p. Default: `1.0`, which is disabled.540 541`repeat_penalty`: Control the repetition of token sequences in the generated text. Default: `1.1`542 543`repeat_last_n`: Last n tokens to consider for penalizing repetition. Default: `64`, where `0` is disabled.544 545`presence_penalty`: Repeat alpha presence penalty. Default: `0.0`, which is disabled.546 547`frequency_penalty`: Repeat alpha frequency penalty. Default: `0.0`, which is disabled.548 549`dry_multiplier`: Set the DRY (Don't Repeat Yourself) repetition penalty multiplier. Default: `0.0`, which is disabled.550 551`dry_base`: Set the DRY repetition penalty base value. Default: `1.75`552 553`dry_allowed_length`: Tokens that extend repetition beyond this receive exponentially increasing penalty: multiplier * base ^ (length of repeating sequence before token - allowed length). Default: `2`554 555`dry_penalty_last_n`: How many tokens to scan for repetitions. Default: `64`, where `0` is disabled.556 557`dry_sequence_breakers`: Specify an array of sequence breakers for DRY sampling. Only a JSON array of strings is accepted. Default: `['\n', ':', '"', '*']`558 559`xtc_probability`: Set the chance for token removal via XTC sampler. Default: `0.0`, which is disabled.560 561`xtc_threshold`: Set a minimum probability threshold for tokens to be removed via XTC sampler. Default: `0.1` (> `0.5` disables XTC)562 563`mirostat`: Enable Mirostat sampling, controlling perplexity during text generation. Default: `0`, where `0` is disabled, `1` is Mirostat, and `2` is Mirostat 2.0.564 565`mirostat_tau`: Set the Mirostat target entropy, parameter tau. Default: `5.0`566 567`mirostat_eta`: Set the Mirostat learning rate, parameter eta.  Default: `0.1`568 569`grammar`: Set grammar for grammar-based sampling.  Default: no grammar570 571`json_schema`: Set a JSON schema for grammar-based sampling (e.g. `{"items": {"type": "string"}, "minItems": 10, "maxItems": 100}` of a list of strings, or `{}` for any JSON). See [tests](../../tests/test-json-schema-to-grammar.cpp) for supported features.  Default: no JSON schema.572 573`seed`: Set the random number generator (RNG) seed.  Default: `-1`, which is a random seed.574 575`ignore_eos`: Ignore end of stream token and continue generating.  Default: `false`576 577`logit_bias`: Modify the likelihood of a token appearing in the generated text completion. For example, use `"logit_bias": [[15043,1.0]]` to increase the likelihood of the token 'Hello', or `"logit_bias": [[15043,-1.0]]` to decrease its likelihood. Setting the value to false, `"logit_bias": [[15043,false]]` ensures that the token `Hello` is never produced. The tokens can also be represented as strings, e.g. `[["Hello, World!",-0.5]]` will reduce the likelihood of all the individual tokens that represent the string `Hello, World!`, just like the `presence_penalty` does. For compatibility with the OpenAI API, a JSON object {"<string or token id>": bias, ...} can also be passed. Default: `[]`578 579`n_probs`: If greater than 0, the response also contains the probabilities of top N tokens for each generated token given the sampling settings. Note that for temperature < 0 the tokens are sampled greedily but token probabilities are still being calculated via a simple softmax of the logits without considering any other sampler settings. Default: `0`580 581`min_keep`: If greater than 0, force samplers to return N possible tokens at minimum. Default: `0`582 583`t_max_predict_ms`: Set a time limit in milliseconds for the prediction (a.k.a. text-generation) phase. The timeout will trigger if the generation takes more than the specified time (measured since the first token was generated) and if a new-line character has already been generated. Useful for FIM applications. Default: `0`, which is disabled.584 585`id_slot`: Assign the completion task to an specific slot. If is -1 the task will be assigned to a Idle slot.  Default: `-1`586 587`cache_prompt`: Re-use KV cache from a previous request if possible. This way the common prefix does not have to be re-processed, only the suffix that differs between the requests. Because (depending on the backend) the logits are **not** guaranteed to be bit-for-bit identical for different batch sizes (prompt processing vs. token generation) enabling this option can cause nondeterministic results. Default: `true`588 589`return_tokens`: Return the raw generated token ids in the `tokens` field. Otherwise `tokens` remains empty. Default: `false`590 591`samplers`: The order the samplers should be applied in. An array of strings representing sampler type names. If a sampler is not set, it will not be used. If a sampler is specified more than once, it will be applied multiple times. Default: `["dry", "top_k", "typ_p", "top_p", "min_p", "xtc", "temperature"]` - these are all the available values.592 593`timings_per_token`: Include prompt processing and text generation speed information in each response.  Default: `false`594 595`return_progress`: Include prompt processing progress in `stream` mode. The progress will be contained inside `prompt_progress` with 4 values: `total`, `cache`, `processed`, and `time_ms`. The overall progress is `processed/total`, while the actual timed progress is `(processed-cache)/(total-cache)`. The `time_ms` field contains the elapsed time in milliseconds since prompt processing started. Default: `false`596 597`sse_ping_interval`: Interval in seconds between SSE comment pings emitted while the stream stays silent, keeping the connection observable during long prompt processing. Overrides the server `--sse-ping-interval` setting for this request, `-1` disables pings. Default: server setting598 599`post_sampling_probs`: Returns the probabilities of top `n_probs` tokens after applying sampling chain.600 601`response_fields`: A list of response fields, for example: `"response_fields": ["content", "generation_settings/n_predict"]`. If the specified field is missing, it will simply be omitted from the response without triggering an error. Note that fields with a slash will be unnested; for example, `generation_settings/n_predict` will move the field `n_predict` from the `generation_settings` object to the root of the response and give it a new name.602 603`lora`: A list of LoRA adapters to be applied to this specific request. Each object in the list must contain `id` and `scale` fields. For example: `[{"id": 0, "scale": 0.5}, {"id": 1, "scale": 1.1}]`. If a LoRA adapter is not specified in the list, its scale will default to `0.0`. Please note that requests with different LoRA configurations will not be batched together, which may result in performance degradation.604 605**Response format**606 607- Note: In streaming mode (`stream`), only `content`, `tokens` and `stop` will be returned until end of completion. Responses are sent using the [Server-sent events](https://html.spec.whatwg.org/multipage/server-sent-events.html) standard. Note: the browser's `EventSource` interface cannot be used due to its lack of `POST` request support.608 609- `completion_probabilities`: An array of token probabilities for each completion. The array's length is `n_predict`. Each item in the array has a nested array `top_logprobs`. It contains at **maximum** `n_probs` elements:610  ```611  {612    "content": "<the generated completion text>",613    "tokens": [ generated token ids if requested ],614    ...615    "probs": [616      {617        "id": <token id>,618        "logprob": float,619        "token": "<most likely token>",620        "bytes": [int, int, ...],621        "top_logprobs": [622          {623            "id": <token id>,624            "logprob": float,625            "token": "<token text>",626            "bytes": [int, int, ...],627          },628          {629            "id": <token id>,630            "logprob": float,631            "token": "<token text>",632            "bytes": [int, int, ...],633          },634          ...635        ]636      },637      {638        "id": <token id>,639        "logprob": float,640        "token": "<most likely token>",641        "bytes": [int, int, ...],642        "top_logprobs": [643          ...644        ]645      },646      ...647    ]648  },649  ```650  Please note that if `post_sampling_probs` is set to `true`:651    - `logprob` will be replaced with `prob`, with the value between 0.0 and 1.0652    - `top_logprobs` will be replaced with `top_probs`. Each element contains:653      - `id`: token ID654      - `token`: token in string655      - `bytes`: token in bytes656      - `prob`: token probability, with the value between 0.0 and 1.0657    - Number of elements in `top_probs` may be less than `n_probs`658 659- `content`: Completion result as a string (excluding `stopping_word` if any). In case of streaming mode, will contain the next token as a string.660- `tokens`: Same as `content` but represented as raw token ids. Only populated if `"return_tokens": true` or `"stream": true` in the request.661- `stop`: Boolean for use with `stream` to check whether the generation has stopped (Note: This is not related to stopping words array `stop` from input options)662- `generation_settings`: The provided options above excluding `prompt` but including `n_ctx`, `model`. These options may differ from the original ones in some way (e.g. bad values filtered out, strings converted to tokens, etc.).663- `model`: The model alias (for model path, please use `/props` endpoint)664- `prompt`: The processed `prompt` (special tokens may be added)665- `stop_type`: Indicating whether the completion has stopped. Possible values are:666  - `none`: Generating (not stopped)667  - `eos`: Stopped because it encountered the EOS token668  - `limit`: Stopped because `n_predict` tokens were generated before stop words or EOS was encountered669  - `word`: Stopped due to encountering a stopping word from `stop` JSON array provided670- `stopping_word`: The stopping word encountered which stopped the generation (or "" if not stopped due to a stopping word)671- `timings`: Hash of timing information about the completion such as the number of tokens `predicted_per_second`672- `tokens_cached`: Number of tokens from the prompt which could be re-used from previous completion673- `tokens_evaluated`: Number of tokens evaluated in total from the prompt674- `truncated`: Boolean indicating if the context size was exceeded during generation, i.e. the number of tokens provided in the prompt (`tokens_evaluated`) plus tokens generated (`tokens predicted`) exceeded the context size (`n_ctx`)675 676 677### POST `/tokenize`: Tokenize a given text678 679*Options:*680 681`content`: (Required) The text to tokenize.682 683`add_special`: (Optional) Boolean indicating if special tokens, i.e. `BOS`, should be inserted.  Default: `false`684 685`parse_special`: (Optional) Boolean indicating if special tokens should be tokenized. When `false` special tokens are treated as plaintext.  Default: `true`686 687`with_pieces`: (Optional) Boolean indicating whether to return token pieces along with IDs.  Default: `false`688 689**Response:**690 691Returns a JSON object with a `tokens` field containing the tokenization result. The `tokens` array contains either just token IDs or objects with `id` and `piece` fields, depending on the `with_pieces` parameter. The piece field is a string if the piece is valid unicode or a list of bytes otherwise.692 693 694If `with_pieces` is `false`:695```json696{697  "tokens": [123, 456, 789]698}699```700 701If `with_pieces` is `true`:702```json703{704  "tokens": [705    {"id": 123, "piece": "Hello"},706    {"id": 456, "piece": " world"},707    {"id": 789, "piece": "!"}708  ]709}710```711 712With input 'á' (utf8 hex: C3 A1) on tinyllama/stories260k713```714{715  "tokens": [716    {"id": 198, "piece": [195]}, // hex C3717    {"id": 164, "piece": [161]} // hex A1718  ]719}720```721 722### POST `/detokenize`: Convert tokens to text723 724*Options:*725 726`tokens`: Set the tokens to detokenize.727 728### POST `/apply-template`: Apply chat template to a conversation729 730Uses the server's prompt template formatting functionality to convert chat messages to a single string expected by a chat model as input, but does not perform inference. Instead, the prompt string is returned in the `prompt` field of the JSON response. The prompt can then be modified as desired (for example, to insert "Sure!" at the beginning of the model's response) before sending to `/completion` to generate the chat response.731 732*Options:*733 734`messages`: (Required) Chat turns in the same format as `/v1/chat/completions`.735 736**Response format**737 738Returns a JSON object with a field `prompt` containing a string of the input messages formatted according to the model's chat template format.739 740### POST `/embedding`: Generate embedding of a given text741 742> [!IMPORTANT]743>744> This endpoint is **not** OAI-compatible. For OAI-compatible client, use `/v1/embeddings` instead.745 746The same as [the embedding example](../embedding) does.747 748This endpoint also supports multimodal embeddings. See the documentation for the `/completions` endpoint for details on how to send a multimodal prompt.749 750*Options:*751 752`content`: Set the text to process.753 754`embd_normalize`: Normalization for pooled embeddings. Can be one of the following values:755```756  -1: No normalization757   0: Max absolute758   1: Taxicab759   2: Euclidean/L2760  >2: P-Norm761```762 763### POST `/reranking`: Rerank documents according to a given query764 765Similar to https://jina.ai/reranker/ but might change in the future.766Requires a reranker model (such as [bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3)) and the `--embedding --pooling rank` options.767 768*Options:*769 770`query`: The query against which the documents will be ranked.771 772`documents`: An array strings representing the documents to be ranked.773 774*Aliases:*775  - `/rerank`776  - `/v1/rerank`777  - `/v1/reranking`778 779*Examples:*780 781```shell782curl http://127.0.0.1:8012/v1/rerank \783    -H "Content-Type: application/json" \784    -d '{785        "model": "some-model",786            "query": "What is panda?",787            "top_n": 3,788            "documents": [789                "hi",790            "it is a bear",791            "The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China."792            ]793    }' | jq794```795 796### POST `/infill`: For code infilling.797 798Takes a prefix and a suffix and returns the predicted completion as stream.799 800*Options:*801 802- `input_prefix`: Set the prefix of the code to infill.803- `input_suffix`: Set the suffix of the code to infill.804- `input_extra`:  Additional context inserted before the FIM prefix.805- `prompt`:       Added after the `FIM_MID` token806 807`input_extra` is array of `{"filename": string, "text": string}` objects.808 809The endpoint also accepts all the options of `/completion`.810 811If the model has `FIM_REPO` and `FIM_FILE_SEP` tokens, the [repo-level pattern](https://arxiv.org/pdf/2409.12186) is used:812 813```txt814<FIM_REP>myproject815<FIM_SEP>{chunk 0 filename}816{chunk 0 text}817<FIM_SEP>{chunk 1 filename}818{chunk 1 text}819...820<FIM_SEP>filename821<FIM_PRE>[input_prefix]<FIM_SUF>[input_suffix]<FIM_MID>[prompt]822```823 824If the tokens are missing, then the extra context is simply prefixed at the start:825 826```txt827[input_extra]<FIM_PRE>[input_prefix]<FIM_SUF>[input_suffix]<FIM_MID>[prompt]828```829 830### **GET** `/props`: Get server global properties.831 832By default, it is read-only. To make POST request to change global properties, you need to start server with `--props`833 834**Response format**835 836```json837{838  "default_generation_settings": {839    "id": 0,840    "id_task": -1,841    "n_ctx": 1024,842    "speculative": false,843    "is_processing": false,844    "params": {845      "n_predict": -1,846      "seed": 4294967295,847      "temperature": 0.800000011920929,848      "dynatemp_range": 0.0,849      "dynatemp_exponent": 1.0,850      "top_k": 40,851      "top_p": 0.949999988079071,852      "min_p": 0.05000000074505806,853      "xtc_probability": 0.0,854      "xtc_threshold": 0.10000000149011612,855      "typical_p": 1.0,856      "repeat_last_n": 64,857      "repeat_penalty": 1.0,858      "presence_penalty": 0.0,859      "frequency_penalty": 0.0,860      "dry_multiplier": 0.0,861      "dry_base": 1.75,862      "dry_allowed_length": 2,863      "dry_penalty_last_n": 64,864      "dry_sequence_breakers": [865        "\n",866        ":",867        "\"",868        "*"869      ],870      "mirostat": 0,871      "mirostat_tau": 5.0,872      "mirostat_eta": 0.10000000149011612,873      "stop": [],874      "max_tokens": -1,875      "n_keep": 0,876      "n_discard": 0,877      "ignore_eos": false,878      "stream": true,879      "n_probs": 0,880      "min_keep": 0,881      "grammar": "",882      "samplers": [883        "dry",884        "top_k",885        "typ_p",886        "top_p",887        "min_p",888        "xtc",889        "temperature"890      ],891      "speculative.n_max": 16,892      "speculative.n_min": 5,893      "speculative.p_min": 0.8999999761581421,894      "timings_per_token": false895    },896    "prompt": "",897    "next_token": {898      "has_next_token": true,899      "has_new_line": false,900      "n_remain": -1,901      "n_decoded": 0,902      "stopping_word": ""903    }904  },905  "total_slots": 1,906  "model_path": "../models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf",907  "chat_template": "...",908  "chat_template_caps": {},909  "modalities": {910    "vision": false911  },912  "media_marker": "<__media_YoNhud46VdDqbuFmKYEO9PY7A4ARzRfg__>",913  "build_info": "b(build number)-(build commit hash)",914  "is_sleeping": false915}916```917 918- `default_generation_settings` - the default generation settings for the `/completion` endpoint, which has the same fields as the `generation_settings` response object from the `/completion` endpoint.919- `total_slots` - the total number of slots for process requests (defined by `--parallel` option)920- `model_path` - the path to model file (same with `-m` argument)921- `chat_template` - the model's original Jinja2 prompt template922- `chat_template_caps` - capabilities of the chat template (see `common/jinja/caps.h` for more info)923- `modalities` - the list of supported modalities924- `is_sleeping` - sleeping status, see [Sleeping on idle](#sleeping-on-idle)925 926### POST `/props`: Change server global properties.927 928To use this endpoint with POST method, you need to start server with `--props`929 930*Options:*931 932- None yet933 934### POST `/embeddings`: non-OpenAI-compatible embeddings API935 936This endpoint supports all poolings, including `--pooling none`. When the pooling is `none`, the responses will contain the *unnormalized* embeddings for *all* input tokens. For all other pooling types, only the pooled embeddings are returned, normalized using Euclidean norm.937 938Note that the response format of this endpoint is different from `/v1/embeddings`.939 940*Options:*941 942Same as the `/v1/embeddings` endpoint.943 944*Examples:*945 946Same as the `/v1/embeddings` endpoint.947 948**Response format**949 950```951[952  {953    "index": 0,954    "embedding": [955      [ ... embeddings for token 0   ... ],956      [ ... embeddings for token 1   ... ],957      [ ... ]958      [ ... embeddings for token N-1 ... ],959    ]960  },961  ...962  {963    "index": P,964    "embedding": [965      [ ... embeddings for token 0   ... ],966      [ ... embeddings for token 1   ... ],967      [ ... ]968      [ ... embeddings for token N-1 ... ],969    ]970  }971]972```973 974### GET `/slots`: Returns the current slots processing state975 976This endpoint is enabled by default and can be disabled with `--no-slots`. It can be used to query various per-slot metrics, such as speed, processed tokens, sampling parameters, etc.977 978If query param `?fail_on_no_slot=1` is set, this endpoint will respond with status code 503 if there is no available slots.979 980**Response format**981 982<details>983<summary>Example with 2 slots</summary>984 985```json986[987  {988    "id": 0,989    "id_task": 135,990    "n_ctx": 65536,991    "speculative": false,992    "is_processing": true,993    "params": {994      "n_predict": -1,995      "seed": 4294967295,996      "temperature": 0.800000011920929,997      "dynatemp_range": 0.0,998      "dynatemp_exponent": 1.0,999      "top_k": 40,1000      "top_p": 0.949999988079071,1001      "min_p": 0.05000000074505806,1002      "top_n_sigma": -1.0,1003      "xtc_probability": 0.0,1004      "xtc_threshold": 0.10000000149011612,1005      "typical_p": 1.0,1006      "repeat_last_n": 64,1007      "repeat_penalty": 1.0,1008      "presence_penalty": 0.0,1009      "frequency_penalty": 0.0,1010      "dry_multiplier": 0.0,1011      "dry_base": 1.75,1012      "dry_allowed_length": 2,1013      "dry_penalty_last_n": 131072,1014      "mirostat": 0,1015      "mirostat_tau": 5.0,1016      "mirostat_eta": 0.10000000149011612,1017      "max_tokens": -1,1018      "n_keep": 0,1019      "n_discard": 0,1020      "ignore_eos": false,1021      "stream": true,1022      "n_probs": 0,1023      "min_keep": 0,1024      "chat_format": "GPT-OSS",1025      "reasoning_format": "none",1026      "reasoning_in_content": false,1027      "generation_prompt": "",1028      "samplers": [1029        "penalties",1030        "dry",1031        "top_k",1032        "typ_p",1033        "top_p",1034        "min_p",1035        "xtc",1036        "temperature"1037      ],1038      "speculative.n_max": 16,1039      "speculative.n_min": 0,1040      "speculative.p_min": 0.75,1041      "timings_per_token": false,1042      "post_sampling_probs": false,1043      "lora": []1044    },1045    "next_token": {1046      "has_next_token": true,1047      "has_new_line": false,1048      "n_remain": -1,1049      "n_decoded": 01050    }1051  },1052  {1053    "id": 1,1054    "id_task": 0,1055    "n_ctx": 65536,1056    "speculative": false,1057    "is_processing": true,1058    "params": {1059      "n_predict": -1,1060      "seed": 4294967295,1061      "temperature": 0.800000011920929,1062      "dynatemp_range": 0.0,1063      "dynatemp_exponent": 1.0,1064      "top_k": 40,1065      "top_p": 0.949999988079071,1066      "min_p": 0.05000000074505806,1067      "top_n_sigma": -1.0,1068      "xtc_probability": 0.0,1069      "xtc_threshold": 0.10000000149011612,1070      "typical_p": 1.0,1071      "repeat_last_n": 64,1072      "repeat_penalty": 1.0,1073      "presence_penalty": 0.0,1074      "frequency_penalty": 0.0,1075      "dry_multiplier": 0.0,1076      "dry_base": 1.75,1077      "dry_allowed_length": 2,1078      "dry_penalty_last_n": 131072,1079      "mirostat": 0,1080      "mirostat_tau": 5.0,1081      "mirostat_eta": 0.10000000149011612,1082      "max_tokens": -1,1083      "n_keep": 0,1084      "n_discard": 0,1085      "ignore_eos": false,1086      "stream": true,1087      "n_probs": 0,1088      "min_keep": 0,1089      "chat_format": "GPT-OSS",1090      "reasoning_format": "none",1091      "reasoning_in_content": false,1092      "generation_prompt": "",1093      "samplers": [1094        "penalties",1095        "dry",1096        "top_k",1097        "typ_p",1098        "top_p",1099        "min_p",1100        "xtc",1101        "temperature"1102      ],1103      "speculative.n_max": 16,1104      "speculative.n_min": 0,1105      "speculative.p_min": 0.75,1106      "timings_per_token": false,1107      "post_sampling_probs": false,1108      "lora": []1109    },1110    "next_token": {1111      "has_next_token": true,1112      "has_new_line": true,1113      "n_remain": -1,1114      "n_decoded": 1361115    }1116  }1117]1118```1119 1120</details>1121 1122### GET `/metrics`: Prometheus compatible metrics exporter1123 1124This endpoint is only accessible if `--metrics` is set.1125 1126In *router mode* the query param `?model={model_id}` has to be set. This endpoint will respond with status code 400 `model name is missing from the request` if not set.1127 1128#### Available metrics1129 1130| Metric | Type | Description |1131| ------ | ---------------------- | ----------- |1132| `llamacpp:prompt_tokens_total` | Counter | Number of prompt tokens processed. |1133| `llamacpp:prompt_seconds_total` | Counter | Prompt process time in seconds. |1134| `llamacpp:prompt_tokens_seconds` | Gauge | Average prompt throughput in tokens/s. |1135| `llamacpp:tokens_predicted_total` | Counter | Number of generation tokens processed. |1136| `llamacpp:tokens_predicted_seconds_total` | Counter | Predict process time in seconds. |1137| `llamacpp:predicted_tokens_seconds` | Gauge | Average generation throughput in tokens/s. |1138| `llamacpp:requests_processing` | Gauge | Number of requests processing. |1139| `llamacpp:requests_deferred` | Gauge | Number of requests deferred. |1140| `llamacpp:n_tokens_max` | Counter | High watermark of the context size observed. |1141| `llamacpp:n_decode_total` | Counter | Total Number of llama_decode() calls. |1142| `llamacpp:n_busy_slots_per_decode` | Gauge | Average number of busy slots per llama_decode() call. |1143| `llamacpp:spec_decode_num_draft_tokens_total` | Counter | Total draft tokens generated (0 when spec-decode is off). |1144| `llamacpp:spec_decode_num_accepted_tokens_total` | Counter | Total draft tokens accepted by the target model (0 when spec-decode is off). |1145| `llamacpp:spec_decode_num_drafts_total` | Counter | Total speculative decoding verification steps (0 when spec-decode is off). |1146| `llamacpp:spec_decode_num_accepted_tokens_per_pos_total` | Counter | Accepted tokens per draft position (labeled `position="N"`; absent when spec-decode is off or before the first completed speculative request). |1147 1148### POST `/slots/{id_slot}?action=save`: Save the prompt cache of the specified slot to a file.1149 1150*Options:*1151 1152`filename`: Name of the file to save the slot's prompt cache. The file will be saved in the directory specified by the `--slot-save-path` server parameter.1153 1154**Response format**1155 1156```json1157{1158    "id_slot": 0,1159    "filename": "slot_save_file.bin",1160    "n_saved": 1745,1161    "n_written": 14309796,1162    "timings": {1163        "save_ms": 49.8651164    }1165}1166```1167 1168### POST `/slots/{id_slot}?action=restore`: Restore the prompt cache of the specified slot from a file.1169 1170*Options:*1171 1172`filename`: Name of the file to restore the slot's prompt cache from. The file should be located in the directory specified by the `--slot-save-path` server parameter.1173 1174**Response format**1175 1176```json1177{1178    "id_slot": 0,1179    "filename": "slot_save_file.bin",1180    "n_restored": 1745,1181    "n_read": 14309796,1182    "timings": {1183        "restore_ms": 42.9371184    }1185}1186```1187 1188### POST `/slots/{id_slot}?action=erase`: Erase the prompt cache of the specified slot.1189 1190**Response format**1191 1192```json1193{1194    "id_slot": 0,1195    "n_erased": 17451196}1197```1198 1199### GET `/lora-adapters`: Get list of all LoRA adapters1200 

Showing the first 1,200 of 2139 lines. Download the file for the rest.