Felipe97/llama-cpp-compiled
01.1k
1# Snapdragon-based devices2 3## Setup4 5The cross-compilation toolchain images are provided by the6[Qualcomm Snapdragon Toolchain registry](https://github.com/snapdragon-toolchain).7These Docker images include the Android NDK, OpenCL SDK, Hexagon SDK, CMake, and the necessary cross-compilers:8 9* **Android toolchain**: `ghcr.io/snapdragon-toolchain/arm64-android:v0.7`10* **Linux toolchain**: `ghcr.io/snapdragon-toolchain/arm64-linux:v0.7`11 12The unified build utility (`scripts/snapdragon/build.py`) automatically pulls13and orchestrates these containers to perform target compilation.14You only need to ensure that Docker (or Docker Desktop on macOS/Windows) is running on your host machine.15Specific setup, build, and installation details for Linux and Windows on Snapdragon platforms are documented in:16* [Linux on Snapdragon guide](linux.md)17* [Windows on Snapdragon guide](windows.md)18 19## How to Build20 21### Using build.py script (Recommended)22 23The easiest way to build llama.cpp is by using the `scripts/snapdragon/build.py` script. It automatically copies the CMake presets,24launches the correct compilation Docker container, builds the libraries and tools,25installs them, and optionally pushes them to your ADB device.26 27Build and deploy for Android target (accepts `android` or `adb` alias):28```29$ ./scripts/snapdragon/build.py --target adb --push30```31 32Build and deploy for Linux target (accepts `linux` or `lnx` alias):33```34$ ./scripts/snapdragon/build.py --target linux:user@host --push35```36 37### Manual CMake Build38 39Alternatively, you can build llama.cpp manually by entering the cross-compilation Docker container and running the CMake commands:40 41```bash42# Start the cross-compilation container manually:43~/src/llama.cpp$ docker run -it --rm -u $(id -u):$(id -g) --volume $(pwd):/workspace --platform linux/amd64 ghcr.io/snapdragon-toolchain/arm64-android:v0.744 45# Inside the container, build the project using presets:46[d]/workspace> cp docs/backend/snapdragon/CMakeUserPresets.json .47 48[d]/workspace> cmake --preset arm64-android-snapdragon-release -B build-snapdragon49Preset CMake variables:50 ANDROID_ABI="arm64-v8a"51 ...52 CMAKE_TOOLCHAIN_FILE="/opt/android-ndk-r28b/build/cmake/android.toolchain.cmake"53 GGML_HEXAGON="ON"54 GGML_OPENCL="ON"55 GGML_OPENMP="OFF"56 HEXAGON_SDK_ROOT="/opt/hexagon/6.6.0.0"57...58-- Including OpenCL backend59-- Including Hexagon backend60...61-- Build files have been written to: /workspace/build-snapdragon62 63[d]/workspace> cmake --build build-snapdragon64...65[144/356] Performing build step for 'htp-v73'66[1/16] Generating htp_iface_skel.c, htp_iface_stub.c, htp_iface.h67[2/16] Building C object CMakeFiles/ggml-htp-v73.dir/hvx-sigmoid.c.obj68[3/16] Building C object CMakeFiles/ggml-htp-v73.dir/htp-dma.c.obj69[4/16] Building C object CMakeFiles/ggml-htp-v73.dir/worker-pool.c.obj70...71-- Installing: /workspace/build-snapdragon/ggml/src/ggml-hexagon/libggml-htp-v73.so72-- Installing: /workspace/build-snapdragon/ggml/src/ggml-hexagon/libggml-htp-v75.so73...74```75 76To generate an installable "package" simply use cmake --install:77 78```79[d]/workspace> cmake --install build-snapdragon --prefix pkg-android/llama.cpp80-- Install configuration: "Release"81-- Installing: /workspace/pkg-android/llama.cpp/lib/libggml-cpu.so82-- Installing: /workspace/pkg-android/llama.cpp/lib/libggml-opencl.so83-- Installing: /workspace/pkg-android/llama.cpp/lib/libggml-hexagon.so84-- Installing: /workspace/pkg-android/llama.cpp/lib/libggml-htp-v73.so85-- Installing: /workspace/pkg-android/llama.cpp/lib/libggml-htp-v75.so86-- Installing: /workspace/pkg-android/llama.cpp/lib/libggml-htp-v79.so87-- Installing: /workspace/pkg-android/llama.cpp/lib/libggml-htp-v81.so88-- Installing: /workspace/pkg-android/llama.cpp/lib/libggml.so89...90-- Installing: /workspace/pkg-android/llama.cpp/bin/llama-bench91-- Installing: /workspace/pkg-android/llama.cpp/bin/llama-cli92...93```94 95## How to Install96 97### Android98 99For this step, your device needs to be configured for on-device development.100Please see https://developer.android.com/studio/debug/dev-options for details.101 102Once ADB is enabled, use `adb push` to install `pkg-android` on the device.103**Note that the toolchain Docker image doesn't have ADB and doesn't set up the ADB bridge. Please use native ADB on the host.**104 105```106~/src/llama.cpp$ adb push pkg-android/llama.cpp /data/local/tmp/107pkg-android/llama.cpp/bin/: 67 files pushed, 0 skipped. 190.2 MB/s (919095042 bytes in 4.607s)108pkg-android/llama.cpp/include/: 19 files pushed, 0 skipped. 20.5 MB/s (255173 bytes in 0.012s)109pkg-android/llama.cpp/lib/: 16 files pushed, 0 skipped. 144.4 MB/s (43801382 bytes in 0.289s)110102 files pushed, 0 skipped. 186.9 MB/s (963151597 bytes in 4.914s)111```112 113At this point, you should also install some models:114 115```116~/src/llama.cpp$ wget https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q4_0.gguf117...1182025-10-11 12:04:52 (10.7 MB/s) - ‘Llama-3.2-1B-Instruct-Q4_0.gguf’ saved [773025920/773025920]119 120~/src/llama.cpp$ adb push Llama-3.2-1B-Instruct-Q4_0.gguf /data/local/tmp/gguf121Llama-3.2-1B-Instruct-Q4_0.gguf: 1 file pushed, 0 skipped. 38.3 MB/s (773025920 bytes in 19.250s)122```123 124### Windows125 126All artifacts are already installed in the `pkg-wos` folder.127To run, you can use the `scripts/snapdragon/run.py` runner script (see details below).128 129## How to Run130 131The easiest way to run llama.cpp cli tools is using the provided `scripts/snapdragon/run.py` wrapper script. This script automatically132maps CLI options to environment variables, resolves executable paths, and runs the command locally, via ADB, or remotely via SSH on the133target device.134 135llama.cpp supports three backends on Snapdragon-based devices: CPU, Adreno GPU (GPUOpenCL), and Hexagon NPU.136You can select which backend(s) to run the model on using the `--device` option of the tool (or `--devices` option in `run.py`).137 138Hexagon NPU behaves as a "GPU" device when it comes to `-ngl` and other offload-related options.139 140Here are some examples of running various llama.cpp tools.141 142Generating a completion with Gemma on Android (relying on default `HTP0:0` device and default thread count `-t 6`):143 144```145~/src/llama.cpp$ ./scripts/snapdragon/run.py --target adb -- llama-completion -m models/gemma-2-2b-it-Q4_0.gguf -f prompts/sample_prompt_1024.txt --jinja -st146...147ggml-hex: Hexagon backend (experimental) : allocating new registry : ndev 1148ggml-hex: Hexagon Arch version v79149ggml-hex: allocating new session: HTP0:0150...151load_tensors: offloading output layer to GPU152load_tensors: offloaded 27/27 layers to GPU153load_tensors: CPU model buffer size = 300.00 MiB154load_tensors: HTP0:0 model buffer size = 1400.26 MiB155...156llama_perf_context_print: prompt eval time = 320.00 ms / 1024 tokens ( 0.31 ms per token, 3200.00 tokens per second)157llama_perf_context_print: eval time = 2100.00 ms / 100 runs ( 21.00 ms per token, 47.62 tokens per second)158```159 160Simple question for Llama-3.2-1B:161 162```163~/src/llama.cpp$ ./scripts/snapdragon/run.py --target android --devices HTP0 -- llama-cli -m Llama-3.2-1B-Instruct-Q4_0.gguf -p "what is the most popular cookie in the world?"164...165ggml-hex: Hexagon backend (experimental) : allocating new registry : ndev 1166ggml-hex: Hexagon Arch version v79167ggml-hex: allocating new session: HTP0168ggml-hex: new session: HTP0 : session-id 0 domain-id 3 uri file:///libggml-htp-v79.so?htp_iface_skel_handle_invoke&_modver=1.0&_dom=cdsp&_session=0 handle 0xb4000072c7955e50169...170load_tensors: offloading output layer to GPU171load_tensors: offloaded 17/17 layers to GPU172load_tensors: CPU model buffer size = 225.49 MiB173load_tensors: HTP0 model buffer size = 504.26 MiB174...175I hope this helps you understand the world's most popular cookies! [end of text]176...177llama_perf_sampler_print: sampling time = 30.08 ms / 487 runs ( 0.06 ms per token, 16191.77 tokens per second)178llama_perf_context_print: load time = 617.94 ms179llama_perf_context_print: prompt eval time = 80.76 ms / 11 tokens ( 7.34 ms per token, 136.21 tokens per second)180llama_perf_context_print: eval time = 9210.59 ms / 475 runs ( 19.39 ms per token, 51.57 tokens per second)181llama_perf_context_print: total time = 9454.92 ms / 486 tokens182llama_perf_context_print: graphs reused = 473183llama_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |184llama_memory_breakdown_print: | - HTP0 (Hexagon) | 2048 = 2048 + ( 0 = 0 + 0 + 0) + 0 |185llama_memory_breakdown_print: | - Host | 439 = 225 + 136 + 77 |186```187 188Op test for MUL_MAT:189 190```191~/src/llama.cpp$ ./scripts/snapdragon/run.py --target adb --devices HTP0:0 -- test-backend-ops -b HTP0:0 -o MUL_MAT192...193Backend 2/3: HTP0:0194Device description: Hexagon195Device memory: 2048 MB (2048 MB free)196MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=1,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0,o=1): OK197MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=2,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0,o=1): OK198MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=3,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0,o=1): OK199```200 201Llama benchmark:202 203```204~/src/llama.cpp$ ./scripts/snapdragon/run.py --target adb --devices HTP0 -- llama-bench -p 128 -n 64 -m Llama-3.2-1B-Instruct-Q4_0.gguf205...206ggml-hex: Hexagon backend (experimental) : allocating new registry : ndev 1207ggml-hex: Hexagon Arch version v79208ggml-hex: allocating new session: HTP0209ggml-hex: new session: HTP0 : session-id 0 domain-id 3 uri file:///libggml-htp-v79.so?htp_iface_skel_handle_invoke&_modver=1.0&_dom=cdsp&_session=0 handle 0xb400007d4b231090210| model | size | params | backend | ngl | threads | n_batch | mmap | test | t/s |211| ---------------| ---------: | -----: | ---------- | --: | ------: | ------: | ---: | ----: | ------------: |212| llama 1B Q4_0 | 729.75 MiB | 1.24 B | HTP | 99 | 4 | 128 | 0 | pp128 | 169.42 ± 1.75 |213| llama 1B Q4_0 | 729.75 MiB | 1.24 B | HTP | 99 | 4 | 128 | 0 | tg64 | 51.54 ± 1.13 |214```215 216## Multi-Device Execution Modes217 218The Hexagon backend supports multiple execution and partitioning modes to accommodate different model sizes, memory219constraints, and single- or multi-NPU hardware topologies:220 221### 1. Single-Device Mode with Dynamic Buffer Mapping222 223Runs the model on a single NPU session (e.g. `HTP0` or `HTP0:0`).224 225A single NPU session provides ~3.5GB of available virtual address space. For models larger than 3.5GB, the backend226automatically maps and unmaps weight buffers during graph execution. This allows large models to run on a single NPU227without manual configuration:228 229```bash230./scripts/snapdragon/run.py --target adb --devices HTP0:0 -- \231 llama-cli -m models/Llama-3.2-3B-Instruct-Q4_0.gguf -ngl 99 -p "Hello"232```233 234### 2. Layer-Split Mode across Virtual Sessions (`HTP0,HTP1,...` or `HTP0:0,HTP0:1,...`)235 236Partitions model layers at load time across multiple virtual sessions hosted on a single physical NPU.237 238Each virtual session acts as an independent backend device from llama.cpp's perspective (similar to multiple GPUs).239Because layers are permanently distributed across sessions, each session's allocated weights remain within its private 3.5GB240address space window, eliminating runtime buffer re-mapping overhead.241 242Here is an example of running the GPT-OSS-20B model on a Snapdragon device using 4 virtual sessions on a single NPU:243 244```bash245./scripts/snapdragon/run.py --target adb \246 --devices HTP0:0,HTP0:1,HTP0:2,HTP0:3 -- \247 llama-cli --load-mode none -m /data/local/tmp/gguf/gpt-oss-20b-Q4_0.gguf -t 4 \248 --ctx-size 8192 --batch-size 128 -ctk q8_0 -ctv q8_0 -fa on -ngl 99 -no-cnv -f surfing.txt249```250 251Log output snippet:252 253```254...255llama_model_loader: - type f32: 289 tensors256llama_model_loader: - type q4_0: 96 tensors257llama_model_loader: - type q8_0: 2 tensors258llama_model_loader: - type mxfp4: 72 tensors259...260load_tensors: offloaded 25/25 layers to GPU261load_tensors: CPU model buffer size = 1182.09 MiB262load_tensors: HTP0:1 model buffer size = 2512.58 MiB263load_tensors: HTP0:3 model buffer size = 2093.83 MiB264load_tensors: HTP0:0 model buffer size = 2931.34 MiB265load_tensors: HTP0:2 model buffer size = 2512.58 MiB266...267llama_perf_context_print: prompt eval time = 3843.67 ms / 197 tokens ( 19.51 ms per token, 51.25 tokens per second)268llama_perf_context_print: eval time = 1686.13 ms / 31 runs ( 54.39 ms per token, 18.39 tokens per second)269llama_perf_context_print: total time = 6266.30 ms / 228 tokens270llama_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |271llama_memory_breakdown_print: | - HTP0:0 (Hexagon) | 2048 = 2048 + ( 0 = 0 + 0 + 0) + 0 |272llama_memory_breakdown_print: | - HTP0:1 (Hexagon) | 2048 = 2048 + ( 0 = 0 + 0 + 0) + 0 |273llama_memory_breakdown_print: | - HTP0:2 (Hexagon) | 2048 = 2048 + ( 0 = 0 + 0 + 0) + 0 |274llama_memory_breakdown_print: | - HTP0:3 (Hexagon) | 2048 = 2048 + ( 0 = 0 + 0 + 0) + 0 |275llama_memory_breakdown_print: | - Host | 1476 = 1208 + 105 + 162 |276```277 278### 3. Tensor-Split Mode across Physical Devices (`HTP0:0,HTP1:0,...`)279 280Distributes model tensors across distinct physical NPU hardware cores using llama.cpp's tensor parallelism281(`--split-mode tensor`).282 283Tensors are partitioned across physical NPUs for parallel execution (proportions are distributed equally by default without284needing an explicit `--tensor-split` option):285 286```bash287./scripts/snapdragon/run.py --target adb \288 --devices HTP0:0,HTP1:0 -- \289 llama-cli -m models/Llama-3.2-3B-Instruct-Q4_0.gguf --split-mode tensor -ngl 99 -p "Hello"290```291 292### 4. Row-Split Multi-Device Mode via Device Grouping (`HTP0[0-1]`)293 294Groups multiple physical NPU cores into a single logical device using bracket notation (`HTP0[0-1]` or `HTP0[0,1]`).295 296Unlike host-level tensor-splitting, row-splitting is executed entirely inside the Hexagon backend:297 298```bash299./scripts/snapdragon/run.py --target adb \300 --devices 'HTP0[0-1]' -- \301 llama-cli -m models/Llama-3.2-3B-Instruct-Q4_0.gguf -ngl 99 -p "Hello"302```303 304You can also combine row-splitting with layer-splitting across multiple grouped devices (e.g. `--devices 'HTP0[0-1],HTP1[2-3]'`305on 4 physical NPUs, or `--devices 'HTP0[0-1:0],HTP1[0-1:1]'` on 2 physical NPUs using virtual sessions 0 and 1).306 307## Environment variables308 309- `GGML_HEXAGON_DEVICES` (default: not set, defaults to HTP0 session)310 Controls which NPU devices and sessions to allocate. Configurable via `--devices` in `run.py`:311 - `N` (single integer): Allocates `N` virtual sessions named `HTP0`, `HTP1`, ..., `HTP<N-1>` on physical NPU 0.312 - `HTP<phys>:<virt>,...`: Comma-separated list of individual devices specifying physical and virtual index:313 - `HTP0:0,HTP0:1`: Two virtual sessions on physical NPU 0 (layer-split on single NPU).314 - `HTP0:0,HTP1:0`: One session on physical NPU 0 and one on physical NPU 1 (tensor-split across physical cores).315 - `HTP<name>[<phys_spec>]`: Device grouping syntax for row-split multi-device execution:316 - `HTP0[0-1]`: A single logical device `HTP0` that groups physical cores 0 and 1.317 - `HTP0[0-1],HTP1[2-3]`: Two layer-split devices across 4 physical NPUs (cores 0-1 and 2-3).318 - `HTP0[0-1:0],HTP1[0-1:1]`: Two layer-split devices across 2 physical NPUs using virtual sessions 0 and 1.319 320- `GGML_HEXAGON_NDEV` (deprecated)321 Replaced by `GGML_HEXAGON_DEVICES`. Controls the number of virtual sessions to allocate on physical NPU `0`.322 Allocates sessions named `HTP0`, `HTP1`, etc.323 324- `GGML_HEXAGON_NHVX=0`325 Controls the number of HVX hardware threads to use. The default is all (actual number varies depending on the hardware version).326 327- `GGML_HEXAGON_HOSTBUF=1` (default: 0, disabled)328 Enables allocating host buffers for debugging. By default, host buffers are disabled.329 330- `GGML_HEXAGON_VERBOSE=1`331 Enables verbose logging of Ops from the backend. Example output:332 333 ```334 ggml-hex: HTP0 graph-compute n_nodes 2335 ggml-hex: HTP0 matmul : blk.27.ffn_up.weight x ffn_norm-27 -> ffn_up-27 : 3072:8192 x 3072:1 -> 8192:1 : q4_0 x f32 -> f32 : HTP0 x HTP0 -> HTP0 : flags 0x1336 ggml-hex: HTP0 matmul : blk.27.ffn_gate.weight x ffn_norm-27 -> ffn_gate-27 : 3072:8192 x 3072:1 -> 8192:1 : q4_0 x f32 -> f32 : HTP0 x HTP0 -> HTP0 : flags 0x3337 ggml-hex: HTP0 graph-compute n_nodes 1338 ggml-hex: HTP0 matmul : blk.27.ffn_down.weight x ffn_gate_par-27 -> ffn_out-27 : 8192:3072 x 8192:1 -> 3072:1 : q4_0 x f32 -> f32 : HTP0 x HTP0 -> HTP0 : flags 0x0339 ggml-hex: HTP0 get-tensor result_output : data 0x7592487000 offset 0 size 513024340 ```341 342- `GGML_HEXAGON_PROFILE=1`343 Enables Op profiling (configurable via `--hex-profile` in `run.py`):344 345 - `1`: Basic profile with per-op `usecs` and `cycles` counters346 - `2`: Extended profile with per-op `usecs`, `cycles` and default PMU counter data347 - `0x1,...,0x8`: Extended profile with per-op `usecs`, `cycles` and custom PMU counter data348 349 The logging output can be saved to a file or piped directly into the post-processing script:350 351 ```bash352 ./scripts/snapdragon/run.py --target adb --hex-profile 1 -- llama-cli ... |& \353 ./scripts/snapdragon/ggml-hexagon-profile.py -354 ```355 356- `GGML_HEXAGON_OPFILTER=regex`357 Filters (disables) Ops matching the regex pattern (configurable via `--hex-opfilter` in `run.py`):358 359 ```bash360 # Disable Flash Attention on Hexagon (falls back to CPU or GPU)361 ./scripts/snapdragon/run.py --target adb --hex-opfilter "FLASH_ATTN_EXT" -- llama-cli ...362 363 # Disable ADD and SUB on Hexagon (fall back to CPU or GPU)364 ./scripts/snapdragon/run.py --target adb --hex-opfilter "ADD|SUB" -- llama-cli ...365 ```366