Felipe97/llama-cpp-compiled

1## Overview2 3> [!IMPORTANT]4> This example and the RPC backend are currently in a proof-of-concept development stage. As such, the functionality is fragile and5> insecure. **Never run the RPC server on an open network or in a sensitive environment!**6 7The `ggml-rpc-server` allows exposing `ggml` devices on a remote host.8The RPC backend communicates with one or several instances of `ggml-rpc-server` and offloads computations to them.9This can be used for distributed LLM inference with `llama.cpp` in the following way:10 11```mermaid12flowchart TD13    rpcb<-->|TCP|srva14    rpcb<-->|TCP|srvb15    rpcb<-.->|TCP|srvn16    subgraph hostn[Host N]17    srvn[ggml-rpc-server]<-.->dev4["CUDA0"]18    srvn[ggml-rpc-server]<-.->dev5["CPU"]19    end20    subgraph hostb[Host B]21    srvb[ggml-rpc-server]<-->dev3["Metal"]22    end23    subgraph hosta[Host A]24    srva[ggml-rpc-server]<-->dev["CUDA0"]25    srva[ggml-rpc-server]<-->dev2["CUDA1"]26    end27    subgraph host[Main Host]28    local["Local devices"]<-->ggml[llama-cli]29    ggml[llama-cli]<-->rpcb[RPC backend]30    end31    style hostn stroke:#66,stroke-width:2px,stroke-dasharray: 5 532    classDef devcls fill:#5B9BD533    class local,dev,dev2,dev3,dev4,dev5 devcls34```35 36By default, `ggml-rpc-server` exposes all available accelerator devices on the host.37If there are no accelerators, it exposes a single `CPU` device.38 39## Usage40 41### Remote hosts42 43On each remote host, build the backends for each accelerator by adding `-DGGML_RPC=ON` to the build options.44For example, to build the `ggml-rpc-server` with support for CUDA accelerators:45 46```bash47mkdir build-rpc-cuda48cd build-rpc-cuda49cmake .. -DGGML_CUDA=ON -DGGML_RPC=ON50cmake --build . --config Release51```52 53When started, the `ggml-rpc-server` will detect and expose all available `CUDA` devices:54 55```bash56$ bin/ggml-rpc-server57ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no58ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no59ggml_cuda_init: found 1 CUDA devices:60  Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes61Starting RPC server v3.0.062  endpoint       : 127.0.0.1:5005263  local cache    : n/a64Devices:65  CUDA0: NVIDIA GeForce RTX 5090 (32109 MiB, 31588 MiB free)66```67 68You can control the set of exposed CUDA devices with the `CUDA_VISIBLE_DEVICES` environment variable or the `--device` command line option. The following two commands have the same effect:69```bash70$ CUDA_VISIBLE_DEVICES=0 bin/ggml-rpc-server -p 5005271$ bin/ggml-rpc-server --device CUDA0 -p 5005272```73 74### Main host75 76On the main host build `llama.cpp` with the backends for the local devices and add `-DGGML_RPC=ON` to the build options.77Finally, when running `llama-cli` or `llama-server`, use the `--rpc` option to specify the host and port of each `ggml-rpc-server`:78 79```bash80$ llama-cli -hf ggml-org/gemma-3-1b-it-GGUF -ngl 99 --rpc 192.168.88.10:50052,192.168.88.11:5005281```82 83By default, llama.cpp distributes model weights and the KV cache across all available devices -- both local and remote -- in proportion to each device's available memory.84You can override this behavior with the `--tensor-split` option and set custom proportions when splitting tensor data across devices.85 86### Local cache87 88The RPC server can use a local cache to store large tensors and avoid transferring them over the network.89This can speed up model loading significantly, especially when using large models.90To enable the cache, use the `-c` option:91 92```bash93$ bin/ggml-rpc-server -c94```95 96By default, the cache is stored in the `$HOME/.cache/llama.cpp/rpc` directory and can be controlled via the `LLAMA_CACHE` environment variable.97 98### RDMA transport99 100The RPC backend can use RDMA instead of TCP for lower latency and higher throughput. The transport is negotiated during the initial handshake -- no changes to command-line usage are required, and the connection falls back to TCP unless both peers can use RDMA.101 102Two providers are supported, each enabled by default when its library is found at build time:103 104- **Linux**: RoCEv2-capable NICs (e.g. Mellanox ConnectX), via `libibverbs`.105- **macOS**: RDMA over Thunderbolt on Apple silicon Macs with Thunderbolt 5, via `librdma`. Requires macOS 26.2 or later, with RDMA enabled once from macOS Recovery via `rdma_ctl enable`. See [TN3205](https://developer.apple.com/documentation/technotes/tn3205-low-latency-communication-with-rdma-over-thunderbolt).106 107RDMA is point-to-point, so each side uses the local device whose GID matches the address the connection was made on. Connect over the RDMA-capable link -- with Thunderbolt, use the peer's Thunderbolt address in `--rpc`; a connection made over another interface stays on TCP.108 109To force plain TCP without rebuilding, set `GGML_RPC_NO_RDMA` on either peer:110```bash111$ GGML_RPC_NO_RDMA=1 bin/ggml-rpc-server112```113 114### Troubleshooting115 116Use the `GGML_RPC_DEBUG` environment variable to enable debug messages from `ggml-rpc-server`:117```bash118$ GGML_RPC_DEBUG=1 bin/ggml-rpc-server119```120 121