Felipe97/llama-cpp-compiled

1# llama-server Development Documentation2 3This document provides an in-depth technical overview of `llama-server`, intended for maintainers and contributors.4 5If you are an end user consuming `llama-server` as a product, please refer to the main [README](./README.md) instead.6 7## Scope of features8 9In-scope types of feature:10 11- Backend:12    - Basic inference features: text completion, embeddings output13    - Chat-oriented features: chat completion, tool calling14    - Third-party API compatibility, e.g. OAI-compat, Anthropic-compat15    - Multimodal input/output16    - Memory management: save/load state, context checkpoints17    - Model management18    - Features that are required by the Web UI19- Frontend:20    - Chat-oriented features, example: basic chat, image upload, edit messages21    - Agentic features, example: MCP22    - Model management23 24Note: For security reasons, features that require reading or writing external files must be **disabled by default**. This covers features like: MCP, model save/load25 26Out-of-scope features:27 28- Backend:29    - Features that require a loop of external API calls, e.g. server-side agentic loop. This is because external API calls in C++ are costly to maintain. Any complex third-party logic should be implemented outside of server code.30    - Features that expose the internal state of the model to the API, example: getting the intermediate activation from API. This is because llama.cpp doesn't support a stable API for doing this, and relying on `eval_callback` can make it complicated to maintain as this API is not intended to be used in multi-sequence setup.31    - Model-specific features. All API calls and features must remain model-agnostic.32- Frontend:33    - Third-party plugins, it is costly to maintain a public plugin API for such features. Instead, users can make their own MCP server for their needs.34    - Customizable themes, it is also costly to maintain. While we do focus on the aesthetic, we try to achieve this by perfecting a small set of themes.35    - Browser-specific features, example: [Chrome's built-in AI API](https://developer.chrome.com/docs/ai/built-in-apis).36 37## Backend38 39### Overview40 41The server supports two primary operating modes:42 43- **Inference mode**: The default mode for performing inference with a single loaded GGUF model.44- **Router mode**: Enables management of multiple inference server instances behind a single API endpoint. Requests are automatically routed to the appropriate backend instance based on the requested model.45 46The core architecture consists of the following components:47 48- `server_context`: Holds the primary inference state, including the main `llama_context` and all active slots.49- `server_slot`: An abstraction over a single “sequence” in llama.cpp, responsible for managing individual parallel inference requests.50- `server_routes`: Middleware layer between `server_context` and the HTTP interface; handles JSON parsing/formatting and request routing logic.51- `server_http_context`: Implements the HTTP server using `cpp-httplib`.52- `server_queue`: Thread-safe queue used by HTTP workers to submit new tasks to `server_context`.53- `server_response`: Thread-safe queue used by `server_context` to return results to HTTP workers.54- `server_response_reader`: Higher-level wrapper around the two queues above for cleaner code.55- `server_task`: Unit of work pushed into `server_queue`.56- `server_task_result`: Unit of result pushed into `server_response`.57- `server_tokens`: Unified representation of token sequences (supports both text and multimodal tokens); used by `server_task` and `server_slot`.58- `server_prompt_checkpoint`: For recurrent (e.g., RWKV) and SWA models, stores snapshots of KV cache state. Enables reuse when subsequent requests share the same prompt prefix, saving redundant computation.59- `server_models`: Standalone component for managing multiple backend instances (used in router mode). It is completely independent of `server_context`.60- `stream_session_manager`: process wide owner of resumable SSE stream sessions, keyed by conversation id. A file-static singleton inside `server-stream.cpp`, driven through `server_stream_session_manager_start/stop`. Backs the replay buffer that lets a client reattach to a generation after an HTTP disconnect. See the "Resumable streaming" section below.61 62```mermaid63graph TD64    API_User <--> server_http_context65    server_http_context <-- router mode --> server_models66    server_http_context <-- inference mode --> server_routes67    server_routes -- server_task --> server_queue68    subgraph server_context69        server_queue --> server_slot70        server_slot -- server_task_result --> server_response71        server_slot[multiple server_slot]72    end73    server_response --> server_routes74```75 76### Batching77 78The server context maintains a single batch shared across all slots. When `update_slots()` is invoked, the system iterates through all active slots to populate this batch. For each slot, either a generated token from the previous decoding step or available prompt tokens are added to the batch.79 80Batching constraints apply: slots can only be batched together if they share compatible configurations. For instance, slots using a specific LoRA adapter can be batched with each other, but not with slots using a different LoRA adapter or no adapter at all.81 82Once the batch reaches capacity or all slots have been processed, `llama_decode` is called to execute the inference. This operation represents the primary computational bottleneck in `update_slots()`.83 84Following decoding, the system either retrieves embeddings or samples the next token using `common_sampler_sample`. If a slot has remaining prompt tokens to process, it yields until the next `update_slots()` iteration.85 86### Thread Management87 88`server_context` runs on a dedicated single thread. Because it is single-threaded, heavy post-processing (especially after token generation) should be avoided, as it directly impacts multi-sequence throughput.89 90Each incoming HTTP request is handled by its own thread managed by the HTTP library. The following operations are performed in HTTP worker threads:91 92- JSON request parsing93- Chat template application94- Tokenization95- Conversion of `server_task_result` into final JSON response96- Error formatting into JSON97- Tracking of partial/incremental responses (e.g., streaming tool calls or reasoning steps)98 99**Best practices to follow:**100 101- All JSON formatting and chat template logic must stay in the HTTP layer.102- Avoid passing raw JSON between the HTTP layer and `server_slot`. Instead, parse everything into native C++ types as early as possible.103 104### Example trace of a request105 106Here is an example trace of an API request for text completion:107 108- A request arrives at the HTTP layer.109- The request is routed to the corresponding handler inside `server_routes`. In this case, `handle_completions_impl` is invoked.110- The handler parses the input request, constructs a new `server_task`, and passes it to `server_res_generator`.111- `server_res_generator` creates a new `task_result_state` for each task:112    - `task_result_state` stays in the HTTP layer, responsible for keeping track of the current state of the response (e.g., parsing tool calls or thinking messages).113    - `server_task` is moved into `server_queue` inside `server_context`.114- `server_context` launches the task by moving it into an available slot (see `launch_slot_with_task()`).115- `update_slot()` processes the task as described in the "Batching" section above.116- Results may be sent using `send_partial_response` or `send_final_response`, which creates a new `server_task_result` and pushes it to the response queue.117- At the same time, `server_res_generator` listens to the response queue and retrieves this response.118- As the response is stateless, `server_res_generator` calls `response->update()` to update the response with the current state.119- `server_res_generator` then calls `response->to_json()` and passes the response to the HTTP layer.120 121### Resumable streaming (SSE replay buffer)122 123By default a streaming generation is bound to its HTTP socket: when the socket drops (refresh, tab close, mobile background, transient network) the generation aborts and the live stream is lost. This feature keeps the generation running server side and lets a client reattach.124 125It is opt in via the `X-Conversation-Id` header on `POST /v1/chat/completions`. Without the header the OAI strict path is unchanged. The conversation id is the only identity end to end (server map key, client localStorage key, route path), with an optional `::model` suffix for direct routing in router mode.126 127The feature lives entirely in `server-stream.{h,cpp}` and rests on three types:128 129- `stream_session`: a bounded ring buffer (4 MiB cap, oldest bytes drop first) plus a condvar. `append` pushes raw SSE bytes, `read_from` drains from any offset and blocks for live bytes or finalize, `finalize` wakes readers, `cancel` sets the flag the producer polls. One conv maps to at most one live session.130- `stream_session_manager`: a file-static singleton (`g_stream_sessions`) inside `server-stream.cpp`, owns all sessions keyed by conv id, enforces the one conv one session invariant via `create_or_replace`, and runs a GC thread that drops completed sessions past their TTL. Exposed to main only through `server_stream_session_manager_start/stop`.131- `stream_pipe_producer` / `stream_pipe_consumer`: the write and read ends. The producer owns the session lifetime and finalizes it on destruction; the consumer is read only and never finalizes, so a reader detaching cannot kill a running generation.132 133The implementation is hidden in `server-stream.cpp` (pimpl). The header exposes only the route handler factories, the `server_res_spipe` response base, `server_stream_conv_id_from_headers` and the GC lifecycle; the session, manager, consumer and the `server_stream_create_spipe` factory stay in the `.cpp`.134 135Producer side: `server_res_generator` extends `server_res_spipe`, which keeps all spipe logic out of the generic `server_http_res`. `set_req` attaches a producer when the header is present, and the wrapped `next` tees each chunk into the ring before the socket, so a chunk lost to a dead wire is already buffered. While attached, `should_stop` ignores peer disconnect: only a `DELETE` stops generation. On an early peer drop, `on_complete` drains the tail into the ring on the http worker.136 137Lifetime safety: the session holds no back reference to the response, so `spipe` is a plain `unique_ptr` touched only by the http worker. `cancel` raises an atomic the producer polls; the producer finalizes the session from its destructor, which also runs `~server_response_reader::stop()` to cancel the generation at the queue level. A `DELETE` stops work by raising the flag and letting the worker unwind.138 139Consumer side: `GET /v1/stream?conv_id=<id>&from=N` opens a `text/event-stream` that replays buffered bytes from offset `N` and blocks for live bytes, so the browser reattaches like a fresh EventSource. An offset below the dropped prefix returns 400.140 141Routes:142 143- `GET /v1/stream?conv_id=<id>&from=N`: replay or live reattach. The id travels in the query string because it can embed a model name containing slashes.144- `POST /v1/streams/lookup` with `{"conversation_ids": [...]}`: returns session status only for ids the caller already owns. There is no listing route, so live sessions cannot be enumerated (an earlier `GET /v1/streams` was removed for exactly this reason).145- `DELETE /v1/stream?conv_id=<id>`: explicit Stop, idempotent (`evict_and_cancel`).146 147Router mode binds the same paths to proxy handlers. A `conv_id -> child` map (`conv_models`), populated when a POST is routed, resolves the owning child in one lookup with no polling. The lookup groups ids per child; GET and DELETE proxy straight to the owner. This loopback REST hop is expected to move to a websocket IPC later, swapping only the transport.148 149Lifecycle: `server_stream_session_manager_start()` runs in main after common init, `server_stream_session_manager_stop()` runs first in `clean_up()` and finalizes every live session so no reader hangs. Reader blocking and the post drop drain both run on httplib worker threads, which block on a condvar rather than spin.150 151| Constant | Value | Role |152| --- | --- | --- |153| `STREAM_SESSION_TTL_SECONDS` | 300 | retention of a completed session before GC |154| `STREAM_SESSION_MAX_BYTES` | 4 MiB | ring cap per session |155| `STREAM_SESSION_GC_INTERVAL_SECONDS` | 60 | GC tick |156| `STREAM_READ_WAKE_INTERVAL_MS` | 200 | read_from wake to recheck should_stop |157| `STREAM_LOOKUP_TIMEOUT_MS` | 250 | router to child loopback budget |158 159```mermaid160graph TD161    Client -- "POST + X-Conversation-Id" --> RG[server_res_generator]162    RG -- attach --> Prod[stream_pipe_producer]163    Prod -- "write, drain on peer drop" --> Sess164    subgraph g_stream_sessions165        Sess[stream_session: ring buffer, 4 MiB]166        GC[GC thread] -- drop after TTL --> Sess167    end168    Sess -- read_from offset --> Cons[stream_pipe_consumer]169    Cons -- "GET /v1/stream?conv_id=id&from=N" --> Client170    DEL[DELETE /v1/stream?conv_id=id] -- evict_and_cancel --> Sess171```172 173The diagram shows the buffer touch points. The live wire (chunks streamed to the original client during a normal generation) is the producer's default output, described under "Producer side" above.174 175### Testing176 177`llama-server` includes an automated test suite based on `pytest`.178 179The framework automatically starts a `llama-server` instance, sends requests, and validates responses.180 181For detailed instructions, see the [test documentation](./tests/README.md).182 183### API for tools184 185This endpoint is intended to be used internally by the Web UI and subject to change or to be removed in the future.186 187**GET /tools**188 189Get a list of tools, each tool has these fields:190- `tool` (string): the ID name of the tool, to be used in POST call. Example: `read_file`191- `display_name` (string): the name to be displayed on UI. Example: `Read file`192- `type` (string): `"server"` for a server tool, or `"mcp"` for a tool exposed by an MCP server193- `permissions` (object): a mapping string --> boolean that indicates the permission required by this tool. This is useful for the UI to ask the user before calling the tool. For now, the only permission supported is `"write"`194- `definition` (object): the OAI-compat definition of this tool195 196**POST /tools**197 198Invoke a tool call, request body is a JSON object with:199- `tool` (string): the name of the tool200- `params` (object): a mapping from argument name (string) to argument value201 202Headers:203- `x-tool-cwd`: optional; if set, use as the CWD for tool; this is not part of tool's params because it's meant to be set by the runtime, not the LLM itself204- `x-tool-runtime`: optional; if set, run the tool inside this isolate instead of on the host. Either `docker-container:<id>` or `podman-container:<id>`, using an already-running container, or `ssh:<target>`, running the tool on a remote host205 206Returns JSON object. There are two response formats (MCP tools use the same two formats: their result content is concatenated into `plain_text_response`, and RPC or tool errors are surfaced as the `error` string):207 208Format 1: Plain text. The text will be placed into a field called `plain_text_response`, example:209 210```json211{212    "plain_text_response": "this is a text response"213}214```215 216The client should extract this value and place it inside message content (note: content is no longer a JSON), example217 218```json219{220    "role": "tool",221    "content": "this is a text response"222}223```224 225Format 2: Normal JSON response, example:226 227```json228{229    "error": "cannot open this file"230}231```232 233That requires `JSON.stringify` when formatted to message content:234 235```json236{237    "role": "tool",238    "content": "{\"error\":\"cannot open this file\"}"239}240```241 242Set `stream: true` in the request body to stream a tool's output as it runs, instead of waiting for it to finish. Only certain tools accept this (for ex. `exec_shell_command`);243returns 404 if tool doesn't support it.244 245Response is SSE stream, one `data: <json>` line per chunk:246 247```json248{"chunk": "hello\n"}249```250 251followed by a final event once the tool returns:252 253```json254{"done": true}255```256 257or, if `invoke()` threw:258 259```json260{"done": true, "error": "..."}261```262 263There is no `[DONE]` sentinel (unlike `/chat/completions`), the stream ends after the `done`264 265### Router mode: how child <--> router communicates266 267Upon spawning a new child process using `subprocess`, both child and router listen to the stdout/stderr (combined)268 269For the direction from child to router:270- Generic messages are logs, it will be forwarded to router's stdout271- Special state update messages are prefixed by `cmd_child_to_router:state:`, followed by a JSON. See `server_models::handle_child_state` for more272 273For the direction from router to child:274- When server sends `cmd_router_to_child:exit`, the child should exit gracefully --> if after `DEFAULT_STOP_TIMEOUT` and the child is still running, force-kill it275 276### Model management API (router mode)277 278Model management API was added via PR [#23976](https://github.com/ggml-org/llama.cpp/pull/23976)279 280The main goal of this API is to allow downloading models and/or removing models from the web UI. It relies on the model cache infrastructure under the hood to manage the list of models dynamically.281 282Instead of building everything from the ground up (like what most AI agents will do when you ask them to implement a similar feature), we built on top of existing, already well-engineered components inside the codebase:283- Model cache infrastructure as mentioned above (`common/download.h`)284- Server response queue (`server-queue.h`). We use this feature to broadcast events to SSE clients.285- Server router thread management (`server-models.h`). We re-use the same thread model that is used for managing subprocess life cycle, except that we don't create a new subprocess, but launch the download right inside the thread.286 287The flow for downloading a new model:288- POST request comes in --> `post_router_models` --> validation289- A new `llama-server` subprocess will be spawned with special `SERVER_CHILD_MODE_DOWNLOAD`290- Child process runs the download and report status back to router via stdin/out291- If a stop request comes in, the router asks the child process to stop (same mechanism as running a model in child process)292- Otherwise, upon completion, we call `load_models()` to refresh the list of models293 294### Sleep mode295 296Sleep mode was initially introduced in PR [#18228](https://github.com/ggml-org/llama.cpp/pull/18228). The main idea is to have:297- `server_queue` keeping track of the idle timeout298- When the timeout is detected, `server_queue` signals to `server_context_impl` that it should go into sleep299- `server_context_impl` frees all `llama_context` and `mtmd_context`300 301Compared to simply exiting the whole process, this approach allows accessing some read-only endpoints during sleep, while also handling wakeup-on-request. Any inference request will wake the server up.302 303Call stack on entering sleeping:304- `server_queue::start_loop` (main thread) sees no task for `idle_sleep_ms` --> `sleeping = true`305- `cb0(true)` --> `server_routes::update_cached_responses`306    - snapshots `/props`, `/models` and metrics; the model is still alive here307- `cb1(true)` --> `server_context_impl::handle_sleeping_state`308    - `callback_state(SERVER_STATE_SLEEPING)` --> reported to router in child mode309    - `destroy()` --> frees `llama_context` and `mtmd_context`310- `condition_tasks.wait` until `req_stop_sleeping`311 312Call stack on waking up:313- `server_res_generator` constructor (HTTP thread) --> `server_queue::wait_until_no_sleep`314    - sets `req_stop_sleeping = true`, then waits until `sleeping == false`315- `server_queue::start_loop` (main thread) wakes up316- `cb1(false)` --> `server_context_impl::handle_sleeping_state`317    - `load_model()`, which then emits `callback_state(SERVER_STATE_READY)`318- `cb0(false)` --> `server_routes::update_cached_responses`319    - nothing to do, the cache is only read during sleep320- `sleeping = false` --> `notify_all` unblocks the HTTP thread, the request is handled as usual321 322Endpoints created with `create_response(true)` (`/health`, `/props`, `/models`, `/metrics`) skip `wait_until_no_sleep`, so they answer from the cached responses instead of waking the server.323 324### Notable Related PRs325 326- Initial server implementation: https://github.com/ggml-org/llama.cpp/pull/1443327- Parallel decoding support: https://github.com/ggml-org/llama.cpp/pull/3228328- Refactor introducing `server_queue` and `server_response`: https://github.com/ggml-org/llama.cpp/pull/5065329- Reranking endpoint: https://github.com/ggml-org/llama.cpp/pull/9510330- Multimodal model support (`libmtmd`): https://github.com/ggml-org/llama.cpp/pull/12898331- Unified KV cache handling: https://github.com/ggml-org/llama.cpp/pull/16736332- Separation of HTTP logic into dedicated files: https://github.com/ggml-org/llama.cpp/pull/17216333- Large-scale code base split into smaller files: https://github.com/ggml-org/llama.cpp/pull/17362334- Introduction of router mode: https://github.com/ggml-org/llama.cpp/pull/17470335- Speculative decoding: https://github.com/ggml-org/llama.cpp/pull/17808 and rework in https://github.com/ggml-org/llama.cpp/pull/17808336- INI presets: https://github.com/ggml-org/llama.cpp/pull/17859 (+ refactoring: https://github.com/ggml-org/llama.cpp/pull/18169)337- Sleeping mode: https://github.com/ggml-org/llama.cpp/pull/18228338- Resumable streaming (SSE replay buffer): https://github.com/ggml-org/llama.cpp/pull/23226339 340 341 342 343## Web UI344 345The project includes a web-based user interface for interacting with `llama-server`. It supports both single-model (`MODEL` mode) and multi-model (`ROUTER` mode) operation.346 347The SvelteKit-based Web UI is introduced in this PR: https://github.com/ggml-org/llama.cpp/pull/14839348 349### Features350 351-   **Chat interface** with streaming responses352-   **Multi-model support** (ROUTER mode) - switch between models, auto-load on selection353-   **Modality validation** - ensures selected model supports conversation's attachments (images, audio)354-   **Conversation management** - branching, regeneration, editing with history preservation355-   **Attachment support** - images, audio, PDFs (with vision/text fallback)356-   **Configurable parameters** - temperature, top_p, etc. synced with server defaults357-   **Dark/light theme**358 359### Tech Stack360 361-   **SvelteKit** - frontend framework with Svelte 5 runes for reactive state362-   **TailwindCSS** + **shadcn-svelte** - styling and UI components363-   **Vite** - build tooling364-   **IndexedDB** (Dexie) - local storage for conversations365-   **LocalStorage** - user settings persistence366 367### Architecture368 369The UI follows a layered architecture:370 371```372Routes → Components → Hooks → Stores → Services → Storage/API373```374 375-   **Stores** - reactive state management (`chatStore`, `conversationsStore`, `modelsStore`, `serverStore`, `settingsStore`)376-   **Services** - stateless API/database communication (`ChatService`, `ModelsService`, `PropsService`, `DatabaseService`)377-   **Hooks** - reusable logic (`useModelChangeValidation`, `useProcessingState`)378 379For detailed architecture diagrams, see [`tools/ui/docs/`](../ui/docs/):380 381-   `high-level-architecture.mmd` - full architecture with all modules382-   `high-level-architecture-simplified.mmd` - simplified overview383-   `data-flow-simplified-model-mode.mmd` - data flow for single-model mode384-   `data-flow-simplified-router-mode.mmd` - data flow for multi-model mode385-   `flows/*.mmd` - detailed per-domain flows (chat, conversations, models, etc.)386 387### Development388 389```sh390# make sure you have Node.js installed391cd tools/ui392npm i393 394# run dev server (with hot reload)395npm run dev396 397# run tests398npm run test399 400# build production bundle401npm run build402```403 404After `public/index.html` has been generated, rebuild `llama-server` as described in the [build](#build) section to include the updated UI.405 406**Note:** The Vite dev server automatically proxies API requests to `http://localhost:8080`. Make sure `llama-server` is running on that port during development.407