Felipe97/llama-cpp-compiled

1# GGML-VirtGPU Backend2 3The GGML-VirtGPU backend enables GGML applications to run machine4learning computations on host hardware while the application itself5runs inside a virtual machine.  It uses host-guest shared memory to6efficiently share data buffers between the two sides.7 8This backend relies on the virtio-gpu, and VirglRenderer API Remoting9(APIR) component. The backend is split into two libraries:10- a GGML implementation (the "remoting frontend"), running in the11  guest and interacting with the virtgpu device12- a VirglRenderer APIR compatible library (the "remoting backend"),13  running in the host and interacting with Virglrenderer and an actual14  GGML device backend.15 16## OS support17 18| OS       | Status            | Backend     | CI testing  | Notes19| -------- | ----------------- | ----------- | ----------- | -----20| MacOS 14 | Supported         | ggml-metal  | X           | Working when compiled on MacOS 1421| MacOS 15 | Supported         | ggml-metal  | X           | Working when compiled on MacOS 14 or MacOS 1522| MacOS 26 | Not tested        |             |             |23| Linux    | Under development | ggml-vulkan | not working | Working locally, CI running into deadlocks24 25 26## Architecture Overview27 28The GGML-VirtGPU backend consists of three main components:29 30```mermaid31graph TD32    %% Nodes33 34 subgraph GuestVM ["Guest VM - Frontend"]35        App([GGML Application<br/>llama.cpp, etc.])36 37        direction TB38        Interface[GGML Backend Interface]39        Comm["GGML-VirtGPU<br/>(hypercalls + shared mem)"]40 41        App --> Interface42        Interface --> Comm43    end44 45    API[virtio-gpu / virglrenderer API]46 47    subgraph HostSystem [Host System - Backend]48        direction TB49        Dispatcher[GGML-VirtGPU-Backend]50        BackendLib[GGML Backend library<br/>Metal / Vulkan / CPU / ...]51 52        Dispatcher --> BackendLib53    end54 55    %% Connections56    Comm --> API57    API --> HostSystem58```59 60### Key Components61 621. **Guest-side Frontend** (`ggml-virtgpu/`): Implements the GGML backend interface and forwards operations to the host632. **Host-side Backend** (`ggml-virtgpu/backend/`): Receives forwarded operations and executes them on actual hardware backends643. **Communication Layer**: Uses virtio-gpu hypercalls and shared memory for efficient data transfer65 66## Features67 68- **Dynamic backend loading** on the host side (CPU, CUDA, Metal, etc.)69- **Zero-copy data transfer** via host-guest shared memory pages70 71## Communication Protocol72 73### Hypercalls and Shared Memory74 75The backend uses two primary communication mechanisms:76 771. **Hypercalls (`DRM_IOCTL_VIRTGPU_EXECBUFFER`)**: Trigger remote execution from guest to host782. **Shared Memory Pages**: Zero-copy data transfer for tensors and parameters79 80#### Shared Memory Layout81 82Each connection uses two shared memory buffers:83 84- **Data Buffer** (24 MiB): For command/response data and tensor transfers85- **Reply Buffer** (16 KiB): For command replies and status information86- **Data Buffers**: Dynamically allocated host-guest shared buffers87  served as GGML buffers.88 89### APIR Protocol90 91The Virglrender API Remoting protocol defines three command types:92 93- `HANDSHAKE`: Protocol version negotiation and capability discovery94- `LOADLIBRARY`: Dynamic loading of backend libraries on the host95- `FORWARD`: API function call forwarding96 97### Binary Serialization98 99Commands and data are serialized using a custom binary protocol with:100 101- Fixed-size encoding for basic types102- Variable-length arrays with size prefixes103- Buffer bounds checking104- Error recovery mechanisms105 106## Supported Operations107 108### Device Operations109- Device enumeration and capability queries110- Memory information (total/free)111- Backend type detection112 113### Buffer Operations114- Buffer allocation and deallocation115- Tensor data transfer (host ↔ guest)116- Memory copying and clearing117 118### Computation Operations119- Graph execution forwarding120 121## Build Requirements122 123### Guest-side Dependencies124- `libdrm` for DRM/virtio-gpu communication125- C++20 compatible compiler126- CMake 3.14+127 128### Host-side Dependencies129- virglrenderer with APIR support (pending upstream review)130- Target backend libraries (libggml-metal, libggml-vulkan, etc.)131 132## Configuration133 134### Environment Variables135 136- `GGML_VIRTGPU_BACKEND_LIBRARY`: Path to the host-side backend library137- `GGML_VIRTGPU_DEBUG`: Enable debug logging138 139### Build Options140 141- `GGML_VIRTGPU`: Enable the VirtGPU backend (`ON` or `OFF`, default: `OFF`)142- `GGML_VIRTGPU_BACKEND`: Build the host-side backend component (`ON`, `OFF` or `ONLY`, default: `OFF`)143 144### System Requirements145 146- VM with virtio-gpu support147- VirglRenderer with APIR patches148- Compatible backend libraries on host149 150## Limitations151 152- **VM-specific**: Only works in virtual machines with virtio-gpu support153- **Host dependency**: Requires properly configured host-side backend154- **Latency**: Small overhead from VM escaping for each operation155- **Shared-memory size**: with the `libkrun` hypervisor, the RAM + VRAM156  addressable memory is limited to 64 GB. So the maximum GPU memory157  will be `64GB - RAM`, regardless of the hardware VRAM size.158 159* This work is pending upstream changes in the VirglRenderer160  project.161  * The backend can be tested with Virglrenderer compiled from source162  using this PR:163  https://gitlab.freedesktop.org/virgl/virglrenderer/-/merge_requests/1590164* This work is pending changes in the VMM/hypervisor running the165  virtual machine, which need to know how to route the newly166  introduced APIR capset.167  * The environment variable `VIRGL_ROUTE_VENUS_TO_APIR=1` allows168    using the Venus capset, until the relevant hypervisors have been169    patched. However, setting this flag breaks the Vulkan/Venus normal170    behavior.171  * The environment variable `GGML_REMOTING_USE_APIR_CAPSET` tells the172    `ggml-virtgpu` backend to use the APIR capset. This will become173    the default when the relevant hypervisors have been patched.174 175* This work focused on improving the performance of llama.cpp running176  on MacOS containers, and is mainly tested on this platform. The177  linux support (via `krun`) is in progress.178 179## See Also180 181- [Development and Testing](VirtGPU/development.md)182- [Backend configuration](VirtGPU/configuration.md)183