Felipe97/llama-cpp-compiled

1# llama.cpp for OpenCL2 3- [llama.cpp for OpenCL](#llamacpp-for-opencl)4  - [Background](#background)5    - [Llama.cpp + OpenCL](#llamacpp--opencl)6  - [OS](#os)7  - [Hardware](#hardware)8    - [Adreno GPU](#adreno-gpu)9  - [DataType Supports](#datatype-supports)10  - [Model Preparation](#model-preparation)11  - [Binary Kernel Library](#binary-kernel-library)12  - [CMake Options](#cmake-options)13  - [Android](#android)14    - [I. Setup Environment](#i-setup-environment)15    - [II. Build llama.cpp](#ii-build-llamacpp)16  - [Windows 11 Arm64](#windows-11-arm64)17    - [I. Setup Environment](#i-setup-environment-1)18    - [II. Build llama.cpp](#ii-build-llamacpp-1)19  - [Linux](#linux)20    - [I. Setup Environment](#i-setup-environment-2)21    - [II. Build llama.cpp](#ii-build-llamacpp-2)22  - [Known Issues](#known-issues)23  - [TODO](#todo)24 25## Background26 27OpenCL (Open Computing Language) is an open, royalty-free standard for cross-platform, parallel programming of diverse accelerators found in supercomputers, cloud servers, personal computers, mobile devices and embedded platforms. OpenCL specifies a programming language (based on C99) for programming these devices and application programming interfaces (APIs) to control the platform and execute programs on the compute devices. Similar to CUDA, OpenCL has been widely used to program GPUs and is supported by most GPU vendors.28 29### Llama.cpp + OpenCL30 31The llama.cpp OpenCL backend is designed to enable llama.cpp on **Qualcomm Adreno GPU** firstly via OpenCL. Thanks to the portabilty of OpenCL, the OpenCL backend can also run on certain Intel GPUs such as those that do not have [SYCL](/docs/backend/SYCL.md) support although the performance is not optimal.32 33## OS34 35| OS      | Status  | Verified                                       |36|---------|---------|------------------------------------------------|37| Android | Support | Snapdragon 8 Gen 3, Snapdragon 8 Elite         |38| Windows | Support | Windows 11 Arm64 with Snapdragon X Elite       |39| Linux   | Support | Ubuntu 22.04 WSL2 with Intel 12700H            |40 41## Hardware42 43### Adreno GPU44 45**Verified devices**46 47| Adreno GPU                            | Status  |48|:-------------------------------------:|:-------:|49| Adreno 750 (Snapdragon 8 Gen 3)       | Support |50| Adreno 810 (Snapdragon 7s Gen 3)      | Support |51| Adreno 830 (Snapdragon 8 Elite)       | Support |52| Adreno 840 (Snapdragon 8 Elite Gen 5) | Support |53| Adreno X1-85 (Snapdragon X Elite)     | Support |54| Adreno X2-90 (Snapdragon X2 Elite)    | Support |55 56> A6x GPUs with a recent driver and compiler are supported; they are usually found in IoT platforms.57However, A6x GPUs in phones are likely not supported due to the outdated driver and compiler.58 59## DataType Supports60 61| DataType               | Status                     |62|:----------------------:|:--------------------------:|63| Q1_0                   | Support                    |64| Q4_0                   | Support                    |65| Q4_1                   | Support                    |66| Q5_0                   | Support                    |67| Q5_1                   | Support                    |68| Q8_0                   | Support                    |69| Q4_K                   | Support                    |70| Q5_K                   | Support                    |71| Q6_K                   | Support                    |72| MXFP4                  | Support                    |73| IQ4_NL                 | Support                    |74 75## Model Preparation76 77Since common quantizations are supported now, it is recommanded to download GGUF models directly from Huggingface.78 79## Binary Kernel Library80 81A prebuilt binary kernel library has been introduced for Adreno GPUs.82It currently targets X2 GPUs (X2-90, X2-85 and X2-45) found in Snapdragon X2 SoC.83The library currently contains kernels for MUL_MAT_ID with Q4_0, Q4_1, Q4_K, MXFP4.84The library must be manually downloaded from https://softwarecenter.qualcomm.com/catalog/item/Adreno_Kernel_Library_GGML.85 86To allow using the kernel library, add `-DGGML_OPENCL_USE_ADRENO_BIN_KERNELS=ON` when configuring with CMake.87Then, extract `adreno-opencl-kernels.dll` from the zip file downloaded from the above URL and put it alongside the executables.88If kernels compatible with the current GPU are found in the library, they will be loaded and used.89 90 91## CMake Options92 93The OpenCL backend has the following CMake options that control the behavior of the backend.94 95| CMake options                        | Default value  | Description                               |96|:------------------------------------:|:--------------:|:------------------------------------------|97| `GGML_OPENCL_EMBED_KERNELS`          | `ON`           | Embed OpenCL kernels into the executable. |98| `GGML_OPENCL_USE_ADRENO_KERNELS`     | `ON`           | Use kernels optimized for Adreno.         |99| `GGML_OPENCL_USE_ADRENO_BIN_KERNELS` | `OFF`          | Allow using binary kernel lib for Adreno. |100 101## Program Binary Cache102 103Compiled `cl_program` binaries are cached on disk, so subsequent runs skip the expensive104compile-from-source step when nothing relevant has changed (kernel source, compile options,105device, driver, or platform version).106 107The cache is controlled with the `GGML_OPENCL_KERNEL_CACHE_DIR` environment variable:108 109| Value                                  | Behavior                                       |110|:---------------------------------------|:-----------------------------------------------|111| unset / empty / `1` / `default`        | Enabled in the platform default cache directory: `%LOCALAPPDATA%\llama.cpp\cl-cache` (Windows), `~/Library/Caches/llama.cpp/cl-cache` (macOS), `<temp dir>/llama.cpp/cl-cache` elsewhere. |112| `0` / `off` / `none` / `disable(d)`    | Disabled.                                      |113| any other value                        | Used verbatim as the cache directory path.     |114 115If the chosen directory cannot be created or used, the cache disables itself for the process116and kernels are compiled from source as usual. Set `GGML_OPENCL_KERNEL_CACHE_DEBUG=1` to117print a HIT/MISS/SAVE trace to stderr.118 119## Android120 121Ubuntu 22.04 is used for targeting Android. Make sure the following tools are accessible from command line,122 123* Git124* CMake 3.29125* Ninja126* Python3127 128### I. Setup Environment129 1301. **Install NDK**131 132```sh133cd ~134wget https://dl.google.com/android/repository/commandlinetools-linux-8512546_latest.zip && \135unzip commandlinetools-linux-8512546_latest.zip && \136mkdir -p ~/android-sdk/cmdline-tools && \137mv cmdline-tools latest && \138mv latest ~/android-sdk/cmdline-tools/ && \139rm -rf commandlinetools-linux-8512546_latest.zip140 141yes | ~/android-sdk/cmdline-tools/latest/bin/sdkmanager "ndk;26.3.11579264"142```143 1442. **Install OpenCL Headers and Library**145 146```sh147mkdir -p ~/dev/llm148cd ~/dev/llm149 150git clone https://github.com/KhronosGroup/OpenCL-Headers && \151cd OpenCL-Headers && \152cp -r CL ~/android-sdk/ndk/26.3.11579264/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include153 154cd ~/dev/llm155 156git clone https://github.com/KhronosGroup/OpenCL-ICD-Loader && \157cd OpenCL-ICD-Loader && \158mkdir build_ndk26 && cd build_ndk26 && \159cmake .. -G Ninja -DCMAKE_BUILD_TYPE=Release \160  -DCMAKE_TOOLCHAIN_FILE=$HOME/android-sdk/ndk/26.3.11579264/build/cmake/android.toolchain.cmake \161  -DOPENCL_ICD_LOADER_HEADERS_DIR=$HOME/android-sdk/ndk/26.3.11579264/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include \162  -DANDROID_ABI=arm64-v8a \163  -DANDROID_PLATFORM=24 \164  -DANDROID_STL=c++_shared && \165ninja && \166cp libOpenCL.so ~/android-sdk/ndk/26.3.11579264/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/aarch64-linux-android167```168 169### II. Build llama.cpp170 171```sh172cd ~/dev/llm173 174git clone https://github.com/ggml-org/llama.cpp && \175cd llama.cpp && \176mkdir build-android && cd build-android177 178cmake .. -G Ninja \179  -DCMAKE_TOOLCHAIN_FILE=$HOME/android-sdk/ndk/26.3.11579264/build/cmake/android.toolchain.cmake \180  -DANDROID_ABI=arm64-v8a \181  -DANDROID_PLATFORM=android-28 \182  -DBUILD_SHARED_LIBS=OFF \183  -DGGML_OPENCL=ON184 185ninja186```187 188## Windows 11 Arm64189 190A Snapdragon X Elite device with Windows 11 Arm64 is used. Make sure the following tools are accessible from command line,191 192* Git193* CMake 3.29194* Clang 19195* Ninja196* Visual Studio 2022197* Powershell 7198* Python199 200Visual Studio provides necessary headers and libraries although it is not directly used for building.201Alternatively, Visual Studio Build Tools can be installed instead of the full Visual Studio.202 203> Note that building using Visual Studio's cl compiler is not supported. Clang must be used. Clang depends on libraries provided by Visual Studio to work. Therefore, Visual Studio must be installed. Alternatively, Visual Studio Build Tools can be installed instead of the full Visual Studio.204 205Powershell 7 is used for the following commands.206If an older version of Powershell is used, these commands may not work as they are.207 208### I. Setup Environment209 2101. **Install OpenCL Headers and Library**211 212```powershell213mkdir -p ~/dev/llm214 215cd ~/dev/llm216git clone https://github.com/KhronosGroup/OpenCL-Headers && cd OpenCL-Headers217mkdir build && cd build218cmake .. -G Ninja `219  -DBUILD_TESTING=OFF `220  -DOPENCL_HEADERS_BUILD_TESTING=OFF `221  -DOPENCL_HEADERS_BUILD_CXX_TESTS=OFF `222  -DCMAKE_INSTALL_PREFIX="$HOME/dev/llm/opencl"223cmake --build . --target install224 225cd ~/dev/llm226git clone https://github.com/KhronosGroup/OpenCL-ICD-Loader && cd OpenCL-ICD-Loader227mkdir build && cd build228cmake .. -G Ninja `229  -DCMAKE_BUILD_TYPE=Release `230  -DCMAKE_PREFIX_PATH="$HOME/dev/llm/opencl" `231  -DCMAKE_INSTALL_PREFIX="$HOME/dev/llm/opencl"232cmake --build . --target install233```234 235### II. Build llama.cpp236 237```powershell238 239mkdir -p ~/dev/llm240cd ~/dev/llm241 242git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp243mkdir build && cd build244 245cmake .. -G Ninja `246  -DCMAKE_TOOLCHAIN_FILE="$HOME/dev/llm/llama.cpp/cmake/arm64-windows-llvm.cmake" `247  -DCMAKE_BUILD_TYPE=Release `248  -DCMAKE_PREFIX_PATH="$HOME/dev/llm/opencl" `249  -DBUILD_SHARED_LIBS=OFF `250  -DGGML_OPENCL=ON251ninja252```253 254## Linux255 256The two steps just above also apply to Linux. When building for linux, the commands are mostly the same as those for PowerShell on Windows, but in the second step they do not have the `-DCMAKE_TOOLCHAIN_FILE` parameter, and then in both steps the backticks are replaced with back slashes.257 258If not installed already, install Git, CMake, Clang, Ninja and Python, then run in the terminal the following:259 260### I. Setup Environment261 2621. **Install OpenCL Headers and Library**263 264```bash265mkdir -p ~/dev/llm266 267cd ~/dev/llm268git clone https://github.com/KhronosGroup/OpenCL-Headers && cd OpenCL-Headers269mkdir build && cd build270cmake .. -G Ninja \271  -DBUILD_TESTING=OFF \272  -DOPENCL_HEADERS_BUILD_TESTING=OFF \273  -DOPENCL_HEADERS_BUILD_CXX_TESTS=OFF \274  -DCMAKE_INSTALL_PREFIX="$HOME/dev/llm/opencl"275cmake --build . --target install276 277cd ~/dev/llm278git clone https://github.com/KhronosGroup/OpenCL-ICD-Loader && cd OpenCL-ICD-Loader279mkdir build && cd build280cmake .. -G Ninja \281  -DCMAKE_BUILD_TYPE=Release \282  -DCMAKE_PREFIX_PATH="$HOME/dev/llm/opencl" \283  -DCMAKE_INSTALL_PREFIX="$HOME/dev/llm/opencl"284cmake --build . --target install285```286 287### II. Build llama.cpp288 289```bash290mkdir -p ~/dev/llm291cd ~/dev/llm292 293git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp294mkdir build && cd build295 296cmake .. -G Ninja \297  -DCMAKE_BUILD_TYPE=Release \298  -DCMAKE_PREFIX_PATH="$HOME/dev/llm/opencl" \299  -DBUILD_SHARED_LIBS=OFF \300  -DGGML_OPENCL=ON301ninja302```303 304## Known Issues305 306- Flash attention does not always improve performance.307- Currently OpenCL backend works on A6xx GPUs with recent drivers and compilers (usually found in IoT platforms).308  However, it does not work on A6xx GPUs found in phones with old drivers and compilers.309 310## TODO311 312- Improve flash attention313- Improve OpenCL C kernels performance314