Felipe97/llama-cpp-compiled

1# Hexagon backend developer details2 3## Backend libraries4 5The Hexagon backend consists of two parts:6 7  - `libggml-hexagon`8    This is the regular CPU-side GGML backend library, either shared or statically linked.9 10  - `libggml-htp-vNN`11    This is the NPU-side (HTP stands for Hexagon Tensor Processor) shared library that contains the Op dispatcher and kernels.12    The correct library is selected automatically at runtime based on the HW version.13 14Here is an example of the build artifacts:15 16```17~/src/llama.cpp$ ls -l pkg-adb/llama.cpp/lib/libggml*18pkg-adb/llama.cpp/lib/libggml-base.so19pkg-adb/llama.cpp/lib/libggml-cpu.so20pkg-adb/llama.cpp/lib/libggml-hexagon.so      <<< CPU library21pkg-adb/llama.cpp/lib/libggml-htp-v73.so      <<< HTP op/kernels for Hexagon v7322pkg-adb/llama.cpp/lib/libggml-htp-v75.so23pkg-adb/llama.cpp/lib/libggml-htp-v79.so24pkg-adb/llama.cpp/lib/libggml-htp-v81.so25```26 27## Memory buffers28 29The Hexagon NPU backend takes advantage of Snapdragon unified memory where all DDR buffers are accessible by CPU, GPU, and NPU.30The NPU has dedicated tightly-coupled memory called VTCM (Vector Tightly-Coupled Memory). VTCM is used for intermediate data (such as31dynamically quantized activations) and streaming buffers (chunks of weight and activation tensors fetched via DMA).32 33## Large model handling34 35Hexagon NPU sessions have a 32-bit virtual address space window of around 3.5GB.36In llama.cpp/GGML, each Hexagon session is mapped to a single GGML backend device (e.g., `HTP0:0`, `HTP0:1`, etc. when using37`GGML_HEXAGON_DEVICES`, or `HTP0`, `HTP1` in legacy mode).38 39To support running models larger than 3.5GB on a single device, the Hexagon backend dynamically maps and unmaps buffers:40- Buffers are allocated in shared DDR (RPCMEM) via file descriptors (`fastrpc_mmap` using `FASTRPC_MAP_FD_DELAYED`).41- Pinned buffers (such as KV cache and active compute buffers) remain mapped throughout execution.42- Inactive weight buffers are dynamically mapped into the NPU session via `HAP_mmap()` during batch buffer preparation43  (`prep_op_bufs()` in `htp/main.c`) and unmapped via `htp_iface_munmap()` when no longer needed by the active batch.44- This dynamic sliding window allows a single NPU session to execute models that exceed the 3.5GB window.45 46Alternatively, users can partition and split the model across multiple virtual sessions or physical NPUs using layer-splitting,47tensor-splitting, or row-splitting modes. For user-facing execution modes and examples, see the48[Snapdragon user guide](README.md#multi-device-execution-modes).49 50## Op and Kernel Development Guidelines51 52Writing high-performance operators for Hexagon requires following specific guidelines.53 54### DDR -> DMA -> VTCM Execution Pipeline55 56- Strongly prefer the `DDR -> DMA -> VTCM -> compute (HVX/HMX) -> VTCM -> DMA -> DDR` data flow.57- Direct HVX reads/writes from/to DDR are less efficient and should only be used as a fallback.58- The DMA queue is a strict FIFO where operations must be pushed and popped in strict order.59- Follow the pipelined multi-buffering sequence properly (typically 2x to 16x buffering) so every push has a corresponding pop:60 61  1. In the prologue, push initial DDR -> VTCM transfers to prime the pipeline.62  2. In the loop body, wait for buffer N via DMA pop, launch HVX/HMX compute on buffer N, push VTCM -> DDR writeback of result N,63     and push DDR -> VTCM prefetch of buffer N+2.64  3. In the epilogue, pop all remaining in-flight transfers to drain the pipeline.65 66- Because every push must be matched by a pop, `dma_queue_flush()` is not required when the pipeline sequence is followed67  properly. Flushing is only used in rare exceptions where a batch of operations is pushed without individual pops.68- Use the DMA queue interface from [`dma-queue.h`](../../../ggml/src/ggml-hexagon/htp/dma-queue.h)69  (`dma_queue_push_ddr_to_vtcm`, `dma_queue_pop`, `dma_queue_push_vtcm_to_ddr`).70  See [`cumsum-ops.c`](../../../ggml/src/ggml-hexagon/htp/cumsum-ops.c) and71  [`act-ops.c`](../../../ggml/src/ggml-hexagon/htp/act-ops.c) for reference implementations.72 73### Avoid Scalar Reads and Writes to VTCM74 75- Access VTCM data using DMA transfers or HVX/HMX vector instructions rather than scalar reads and writes.76 77### Avoid Scalar Division in Inner Loops78 79- Hexagon cores do not have hardware division instructions.80- For recurring divisions across iterations or threads, use `fastdiv` from81  [`hex-fastdiv.h`](../../../ggml/src/ggml-hexagon/htp/hex-fastdiv.h) with precomputed divisors (such as82  `octx->ctx->mdev.count_div` or `octx->n_threads_div`).83- Do not call `init_fastdiv_values()` for single-use divisions; use standard compiler division (`/`) instead.84 85### Host-Side Precomputation via `kernel_params`86 87- Precompute tensor shapes, strides, scale conversions, tiling layouts, and validation checks on the host CPU during graph88  preparation in [`ggml-hexagon.cpp`](../../../ggml/src/ggml-hexagon/ggml-hexagon.cpp).89- Pack precomputed parameters into the operator's fixed `kernel_params` structure in `htp_op_node` (such as90  `htp_mm_kernel_params`, `htp_unary_kernel_params`, `htp_fa_kernel_params`, `htp_get_rows_kernel_params`).91- The NPU executes directly using `octx->kernel_params` without redundant runtime metadata extraction or validation.92- **Strict Host-Kernel Alignment**:93  - Verify that parameters calculated by the host CPU are strictly honored by the NPU kernel.94  - Ensure the kernel does not ignore host-computed fields (for example, falling back to `octx->n_threads` instead of95    using `kparams->n_threads`, or ignoring precomputed `tasks_per_thread` and chunk counts).96  - Both human developers and coding agents must audit both sides of the interface: ensure fields populated in `kernel_params`97    in [`ggml-hexagon.cpp`](../../../ggml/src/ggml-hexagon/ggml-hexagon.cpp) are actively and consistently utilized by the98    corresponding operator entry point and worker threads in `htp/*-ops.c`.99 100### Tracing Instrumentation101 102- All kernels must include trace events for performance profiling and timeline visualization in Perfetto103  ([`hex-profile.h`](../../../ggml/src/ggml-hexagon/htp/hex-profile.h)).104- Surround compute sections with `htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) info)` and105  `htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) info)`.106- Use specific event types for major phases:107  - `HTP_TRACE_EVT_HVX_COMP`: Vector compute execution.108  - `HTP_TRACE_EVT_DMA`: DMA transfer wait or poll cycles.109  - `HTP_TRACE_EVT_FENCE`: Multi-device fence barrier synchronization.110  - `HTP_TRACE_EVT_L2FLUSH`: L2 cache cleaning operations.111- Pass meaningful progress metrics (such as row index, chunk index, or token index) in the 16-bit `info` parameter.112 113### Work Queue and Threading114 115- Distribute parallel work across NPU worker threads using the thread pool work queue:116 117  ```c118  work_queue_run(ctx->work_queue, worker_func, &op_ctx, n_threads);119  ```120 121- Keep worker functions independent and re-entrant. Worker threads should only operate on their designated chunk of rows or elements.122 123### Avoid Redundant Defensive NULL Checks124 125- Do not add defensive NULL checks or assertions for internal framework pointers or required graph operands and outputs.126  Internal pointers include `ctx`, `octx`, local context structs like `*ctx`, `kparams`, and worker callback `data`.127- These pointers are architectural invariants during kernel execution and host-side graph preparation.128  Graph compute receives allocated nodes with valid required `node->src[N]` and `node->data` pointers.129- Do not turn an invariant violation into an unsupported operation or missed fusion.130  Checks such as `if (!octx || !octx->ctx)` clutter the code, obscure intent, and hide upstream errors.131- **Distinction**: `octx->src[N]` pointers *can* be NULL by design and must be checked when optional.132  Examples include attention masks, optional bias or weights in fused kernels, and frequency factors.133 134### Multiline Macro Formatting135 136- Keep trailing backslashes in multiline `#define` macros cleanly aligned to a consistent column.137- Avoid trailing whitespace after macro backslashes.138- Use [`scripts/snapdragon/ggml-hexagon-align-macros.py`](../../../scripts/snapdragon/ggml-hexagon-align-macros.py) to inspect, diff,139  or automatically align macro definitions across Hexagon kernel sources:140 141  ```bash142  # Check for misaligned macros143  python3 scripts/snapdragon/ggml-hexagon-align-macros.py ggml/src/ggml-hexagon/htp/144 145  # Fix misaligned macros in-place146  python3 scripts/snapdragon/ggml-hexagon-align-macros.py --fix ggml/src/ggml-hexagon/htp/147  ```148 149## Multi-Device Partitioning (mdev)150 151Multi-device (mdev) mode enables row-level tensor parallel execution across multiple physical NPU cores or virtual NPU152sessions.153 154### 128-Byte Cache Line Alignment155 156- Shared tensor buffers reside in DDR (RPCMEM) with a 128-byte cache line granularity157  (`HEX_L2_LINE_SIZE` = 128 bytes, `HTP_TENSOR_MDEV_LINE_SIZE`).158- **Rule**: Multi-device work partitions must align destination write regions to 128-byte cache line boundaries so distinct159  devices never share or overwrite the same cache line.160 161### Partitioning Helpers in `htp-tensor.h`162 163Common partitioning logic is factored into reusable inline helpers in164[`htp-tensor.h`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h):165 1661. [`htp_tensor_mdev_rows_per_chunk`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L67):167   Determines the minimum number of rows per chunk so that the chunk byte size is a multiple of 128 bytes:168 169   ```170   rows_per_chunk = 128 / hex_gcd_u32(row_size, 128)171   ```172 173   If row stride `nb[1]` is already a multiple of 128 bytes, `rows_per_chunk = 1`.174   Returns `false` if the tensor cannot be safely row-partitioned (such as unaligned base pointer, permuted layout,175   or non-128-byte aligned outer strides).176 1772. [`htp_tensor_mdev_partition`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L94):178   Calculates the per-device work range `struct htp_tensor_mdev_range { uint32_t start; uint32_t count; }` given179   `total_units`, `units_per_chunk`, `mdev_idx`, `mdev_count`, and the precomputed `mdev_count_div`.180   Handles chunk distribution across devices, assigns remainder units to the last device, and automatically triggers181   single-device fallback when partitioning is unsafe.182 183### Row-Partitioned Operators184 185For row-wise operators186(such as activations in [`act-ops.c`](../../../ggml/src/ggml-hexagon/htp/act-ops.c),187binary ops in [`binary-ops.c`](../../../ggml/src/ggml-hexagon/htp/binary-ops.c),188unary ops in [`unary-ops.c`](../../../ggml/src/ggml-hexagon/htp/unary-ops.c), and189sameshape copies in [`cpy-ops.c`](../../../ggml/src/ggml-hexagon/htp/cpy-ops.c)):190 191```c192const uint32_t total_rows   = ne01 * ne02 * ne03;193const size_t   dst_row_size = dst->ne[0] * elem_size;194 195uint32_t row_start = 0;196uint32_t nrows     = total_rows;197 198if (octx->ctx->mdev.count > 1) {199    uint32_t rows_per_chunk = 0;200    htp_tensor_mdev_rows_per_chunk(dst, elem_size, (uint32_t) dst_row_size, &rows_per_chunk);201    const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(202        total_rows, rows_per_chunk, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);203    row_start = range.start;204    nrows     = range.count;205}206 207if (nrows == 0) {208    return HTP_STATUS_OK;209}210```211 212### Element-Partitioned Operators213 214For flat element-wise operations (such as reshape copies in215[`cpy-ops.c`](../../../ggml/src/ggml-hexagon/htp/cpy-ops.c)):216- Partition total linear elements N = ne0 * ne1 * ne2 * ne3 in 128-byte cache line chunks (`elems_per_line = (elem_size == 4) ? 32 : 64`).217- Requires strict 1D contiguity:218  [`htp_tensor_is_contiguous(dst, elem_size)`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L28)219  and 128-byte aligned destination pointer220  [`htp_tensor_mdev_data_aligned(dst)`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L47).221- If contiguous and aligned, pass `elems_per_line` to222  [`htp_tensor_mdev_partition`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L94);223  otherwise pass 0 to trigger Device 0 fallback.224 225### Single-Device Fallback (Device 0)226 227- Fallback to Device 0 (`mdev.idx == 0`) when partitioning would cause cache line tearing or when work cannot be evenly distributed.228- Triggers:229  1. Destination tensor cannot be safely partitioned (`rows_per_chunk == 0` or non-contiguous/unaligned buffer).230  2. Total aligned chunks < `mdev_count`.231- Device 0 processes the entire tensor `[0, total_units)`.232- Devices 1 ... N-1 receive `count = 0` and return `HTP_STATUS_OK` immediately.233 234### Flatten Outer Dimensions Globally235 236- **Never partition solely on `ne01` (dimension 1).**237- Partitioning only on `ne01` repeats the device boundary across every 2D slice (`ne02`, `ne03`). If each 2D slice is small,238  false sharing occurs repeatedly throughout the tensor.239- Always flatten outer dimensions globally: `total_rows = ne01 * ne02 * ne03` and partition once across the combined row space.240 241### Stateless Starting Coordinates242 243- Do not use incremental state variables across slices that assume the thread or device starts at index 0.244- Precompute starting multidimensional coordinates at `r = row_start` (or `e = elem_start`) once using `fastdiv`.245- In inner loops, step base pointers directly (`ptr += stride`) or reset/wrap coordinates explicitly (`if (++i01 == ne01) { ... }`).246 247### Clean Range Encapsulation248 249- Initialize single-device default ranges at declaration:250 251  ```c252  uint32_t row_start = 0;253  uint32_t nrows     = total_rows;254  ```255 256- Encapsulate all multi-device logic inside `if (octx->ctx->mdev.count > 1)`. If the block is omitted or compiled out,257  the operator runs standard single-device execution untouched.258- Do not propagate `mdev_` prefixes to worker functions or context structs. Worker threads are device-agnostic and259  should only receive standard range parameters (`ctx.row_start`, `ctx.nrows`).260- In worker threads, calculate row intervals using standard arithmetic:261 262  ```c263  const uint32_t ir0 = ctx->row_start + dr * ith;264  const uint32_t ir1 = MIN(ir0 + dr, ctx->row_start + ctx->nrows);265  ```266 267  In single-device mode (`row_start == 0`), this naturally simplifies to `dr * ith` and `MIN(ir0 + dr, ctx->nrows)` with zero overhead.268 269## Multi-Device Synchronization270 271Multi-device execution synchronizes worker sessions across devices using explicit barriers and tensor cache flushing.272 273### Synchronization Fence Protocol274 275Multi-device execution synchronizes worker sessions through atomic fence slots and barriers defined in276[`htp-fence.h`](../../../ggml/src/ggml-hexagon/htp/htp-fence.h):277 278```279[NPU Session 0]                              [NPU Session 1]280       |                                            |281  (Input Prep)                                 (Input Prep)282       |                                            |283  Pre-Op Barrier ----------------------------- Pre-Op Barrier284  (mdev_sync_fence)                            (mdev_sync_fence)285       |                                            |286  Kernel Execution                             Kernel Execution287  (Output Slice 0)                             (Output Slice 1)288       |                                            |289  Tensor Cache Flush                           Tensor Cache Flush290  (htp_tensor_flush_all)                       (htp_tensor_flush_all)291       |                                            |292  Post-Op/Batch Barrier ---------------------- Post-Op/Batch Barrier293  (htp_mdev_group_barrier)                     (htp_mdev_group_barrier)294       |                                            |295  Return Response to Host                      Return Response to Host296```297 298### Atomic Fence Slots and Cache Invalidation299 300- Fence synchronization operates on dedicated RPCMEM shared memory mapped across all participating sessions (`ctx->mdev.fence_base`).301- Each device owns a dedicated 128-byte cache-line aligned fence slot:302 303  ```c304  atomic_uint * my_fence = htp_mdev_fence_slot(fence_base, mdev_idx);305  ```306 307- **Writing to fence ([`htp_fence_write`](../../../ggml/src/ggml-hexagon/htp/htp-fence.h#L18))**:308  Stores `seq` and `status`, issues a `syncht` thread synchronization barrier, and flushes/invalidates the line309  using `Q6_dccleaninva_A(fence)`.310- **Reading from peer fence ([`htp_fence_read`](../../../ggml/src/ggml-hexagon/htp/htp-fence.h#L26))**:311  Executes `Q6_dccleaninva_A(fence)` and `syncht` before reading atomic values to ensure fresh data from DDR.312 313### Deterministic Monotonic Sequence Numbers314 315- Barrier fences use monotonically increasing sequence numbers:316 317  ```c318  const uint32_t seq = ++ctx->mdev.fence_seq;319  ```320 321- Comparing sequence numbers with signed arithmetic `(int32_t)(peer_seq - seq) >= 0` prevents race conditions or322  misaligned barrier arrivals across iterations.323- If any peer reports an error status (`peer_status > HTP_STATUS_OK`), the barrier propagates the error and unblocks immediately.324 325### Tensor Cache Flush and Pipeline Completion326 327- In the kernel, ensure all pushed DMA operations have been popped in strict FIFO order to drain the queue.328- Use [`htp_tensor_flush_all()`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h) to flush specific dirty tensors back to DDR:329  - [`htp_tensor_flush_all()`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h) flushes only modified tensor address ranges,330    ensuring peer devices and the host CPU observe consistent data in DDR.331- Never signal completion before all DMA transfers are drained and dirty tensor flushes have completed.332 333