Felipe97/llama-cpp-compiled

1# llama.cpp for AMD ZenDNN2 3> [!WARNING]4> **Note:** ZenDNN is **not** the same as zDNN.5> - **ZenDNN** (this page): AMD's deep learning library for AMD EPYC CPUs6> - **zDNN**: IBM's Deep Neural Network acceleration library for IBM Z & LinuxONE Mainframes ([see zDNN documentation](zDNN.md))7 8- [Background](#background)9- [OS](#os)10- [Hardware](#hardware)11- [Supported Operations](#supported-operations)12- [DataType Supports](#datatype-supports)13- [Linux](#linux)14- [Environment Variable](#environment-variable)15- [Performance Optimization](#performance-optimization)16- [Known Issues](#known-issues)17- [TODO](#todo)18 19## Background20 21**ZenDNN** (Zen Deep Neural Network Library) is AMD's high-performance deep learning inference library optimized for AMD EPYC™ CPUs. It provides optimized implementations of key deep learning primitives and operations, delivering significant performance improvements for neural network workloads on AMD Zen-based processor architectures.22 23**Llama.cpp + ZenDNN**24 25The llama.cpp ZenDNN backend leverages AMD's optimized matrix multiplication primitives to accelerate inference on AMD CPUs. It utilizes ZenDNN's **LowOHA (Low Overhead Hardware Accelerated)** MatMul operator for efficient GEMM operations with minimal execution overhead, built-in weight caching, and direct access to backend libraries (AOCL DLP, LibXSMM, OneDNN).26 27For more information about ZenDNN, visit: https://www.amd.com/en/developer/zendnn.html28 29## OS30 31| OS      | Status  | Verified                                       |32|:-------:|:-------:|:----------------------------------------------:|33| Linux   | Support | Ubuntu 20.04, 22.04, 24.04                     |34 35For the latest list of supported operating systems, see the [ZenDNN Supported OS](https://github.com/amd/ZenDNN/blob/a18adf8c605fb5f5e52cefd7eda08a7b18febbaf/README.md#15-supported-os).36 37## Hardware38 39### AMD CPUs40 41**Recommended Processors**42 43ZenDNN is optimized for AMD EPYC™ processors and AMD Ryzen™ processors based on "Zen" microarchitecture and newer.44 45| CPU Family                    | Status  | Notes                              |46|:-----------------------------:|:-------:|:----------------------------------:|47| AMD EPYC™ 9005 Series (Turin) | Support | 5th Gen - Zen 5 architecture       |48| AMD EPYC™ 9004 Series (Genoa) | Support | 4th Gen - Zen 4 architecture       |49| AMD EPYC™ 7003 Series (Milan) | Support | 3rd Gen - Zen 3 architecture       |50| AMD Ryzen™ AI MAX (Strix Halo)| Support | High-performance mobile processors |51 52*Notes:*53 54- Best performance is achieved on AMD EPYC™ processors with high core counts (e.g., EPYC 9005 series).55- ZenDNN leverages AMD's advanced CPU features including AVX2 and AVX-512 instruction sets.56- For optimal performance, ensure your system has sufficient memory bandwidth.57 58## Supported Operations59 60The ZenDNN backend accelerates **matrix multiplication (MUL_MAT)** and **expert-based matrix multiplication (MUL_MAT_ID)** operations. Other operations are handled by the standard CPU backend.61 62| Operation    | Status  | Notes                                          |63|:-------------|:-------:|:----------------------------------------------:|64| MUL_MAT      | Support | Accelerated via ZenDNN LowOHA MatMul           |65| MUL_MAT_ID   | Support | Accelerated via ZenDNN LowOHA MatMul (MoE)     |66 67*Note:* Since MUL_MAT and MUL_MAT_ID are accelerated, models will benefit most from ZenDNN when matrix multiplications dominate the computational workload (which is typical for transformer-based LLMs and Mixture-of-Experts models).68 69## DataType Supports70 71| DataType               | Status  | Notes                                         |72|:----------------------:|:-------:|:---------------------------------------------:|73| FP32                   | Support | Full precision floating point                 |74| BF16                   | Support | BFloat16 (best performance on Zen 4/Zen 5)    |75| Q8_0                   | Support | 8-bit quantized weights via [dynamic quantization](https://github.com/amd/ZenDNN/blob/main/docs/operator/lowoha_matmul_operator.md) |76 77*Notes:*78 79- **BF16** provides best performance on Zen 4 and Zen 5 EPYC™ processors (Genoa, Turin).80- **Q8_0** is available for quantized model weights since ZenDNN supports dynamic quantization [LowOHA MatMul operator](https://github.com/amd/ZenDNN/blob/main/docs/operator/lowoha_matmul_operator.md).81- Other quantization formats fall back to the standard CPU backend unless explicitly supported by the ZenDNN backend.82 83## Linux84 85### I. Setup Environment86 87You have two options to set up ZenDNN:88 89#### Option 1: Automatic Download and Build (Recommended)90 91CMake will automatically download and build ZenDNN for you:92 93```sh94# Build llama.cpp - ZenDNN will be automatically downloaded and built95cmake -B build -DGGML_ZENDNN=ON -DCMAKE_BUILD_TYPE=Release96cmake --build build --config Release -j $(nproc)97```98 99No manual ZenDNN installation required. CMake will handle everything automatically.100 101#### Option 2: Use Custom ZenDNN Installation102 103If you want to build ZenDNN yourself or use a specific version:104 105**Step 1: Build ZenDNN from source**106 107```sh108# Clone ZenDNN repository109git clone https://github.com/amd/ZenDNN.git110cd ZenDNN111 112# Build and install (requires CMake >= 3.25)113mkdir build && cd build114cmake ..115cmake --build . --target all116```117 118Default installation path: `ZenDNN/build/install`119 120**For detailed build instructions**, refer to the [ZenDNN README](https://github.com/amd/ZenDNN/blob/a18adf8c605fb5f5e52cefd7eda08a7b18febbaf/README.md).121 122**Step 2: Build llama.cpp with custom ZenDNN path**123 124```sh125# Using environment variable126export ZENDNN_ROOT=/path/to/ZenDNN/build/install127cmake -B build -DGGML_ZENDNN=ON -DCMAKE_BUILD_TYPE=Release128cmake --build build --config Release -j $(nproc)129 130# OR specify path directly in CMake131cmake -B build -DGGML_ZENDNN=ON -DZENDNN_ROOT=/path/to/ZenDNN/build/install -DCMAKE_BUILD_TYPE=Release132cmake --build build --config Release -j $(nproc)133```134 135### II. Run the Server136 137#### 1. Download Model138 139Download LLaMA 3.1 8B Instruct BF16 model:140 141```sh142# Download from Hugging Face143huggingface-cli download meta-llama/Llama-3.1-8B-Instruct-GGUF --local-dir models/144```145 146You can also use a Q8_0 GGUF model:147 148```sh149# Download a Q8_0 GGUF model from Hugging Face150huggingface-cli download meta-llama/Llama-3.1-8B-Instruct-GGUF \151    Llama-3.1-8B-Instruct-Q8_0.gguf \152    --local-dir models/153```154 155#### 2. Start Server156 157Run llama.cpp server with ZenDNN acceleration:158 159```sh160# Set optimal configuration161export ZENDNNL_MATMUL_ALGO=1    # Blocked AOCL DLP algo for best performance162 163# Start server164./build/bin/llama-server \165    -m models/Llama-3.1-8B-Instruct.BF16.gguf \166    --host 0.0.0.0 \167    --port 8080 \168    -t 64169```170 171Access the server at `http://localhost:8080`.172 173**Performance tips**:174- Use `ZENDNNL_MATMUL_ALGO=1` for optimal performance175- For NUMA systems: `numactl --cpunodebind=0 --membind=0 ./build/bin/llama-server ...`176 177## Environment Variable178 179For environment variables related to ZenDNN, refer to the [ZenDNN Environment Variables Documentation](https://github.com/amd/ZenDNN/blob/a18adf8c605fb5f5e52cefd7eda08a7b18febbaf/docs/runtime_env.md).180 181### Performance Optimization182 183ZenDNN's LowOHA MatMul supports multiple backend algorithms. For **best performance**, use the **Blocked AOCL DLP** algorithm:184 185```sh186export ZENDNNL_MATMUL_ALGO=1    # Blocked AOCL DLP algo (recommended)187```188 189For more details on available algorithms, see the [ZenDNN MatMul Algorithm Documentation](https://github.com/amd/ZenDNN/blob/a18adf8c605fb5f5e52cefd7eda08a7b18febbaf/docs/runtime_env.md#algorithm-details).190 191### Q8_0 Performance Notes192 193Q8_0 support is mainly beneficial for prompt processing / prefill workloads where large matrix multiplications dominate execution. Token generation performance may remain close to the standard CPU backend depending on the model, batch size, number of threads, and CPU topology.194 195### Profiling and Debugging196 197For detailed profiling and logging options, refer to the [ZenDNN Logging Documentation](https://github.com/amd/ZenDNN/blob/a18adf8c605fb5f5e52cefd7eda08a7b18febbaf/docs/logging.md).198 199## Known Issues200 201- **Limited operation support**: Currently matrix multiplication (MUL_MAT) and expert-based matrix multiplication (MUL_MAT_ID) are accelerated via ZenDNN. Other operations fall back to the standard CPU backend. Future updates may expand supported operations.202- **BF16 support**: BF16 operations require AMD Zen 4 or Zen 5 architecture (EPYC 9004/9005 series). On older CPUs, operations will use FP32.203- **Q8_0 support scope**: Q8_0 acceleration is available for supported matrix multiplication paths. Other quantization formats still fall back to the standard CPU backend.204- **NUMA awareness**: For multi-socket systems, manual NUMA binding may be required for optimal performance.205 206## Q&A207 208**Q: How do I verify that ZenDNN backend is being used?**209 210A: Check the log output when running llama.cpp. You should see messages indicating the ZenDNN backend is initialized. You can also check the backend name in the output.211 212**Q: What performance improvement can I expect?**213 214A: Performance gains vary depending on the model size, batch size, and CPU architecture. On AMD EPYC processors, you can typically expect 1.1x-2x speedup compared to standard CPU inference for matrix multiplication operations.215 216**Q: Can I use ZenDNN on non-AMD processors?**217 218A: ZenDNN is optimized specifically for AMD processors. While it may work on other x86-64 CPUs, performance benefits are only guaranteed on AMD Zen-based architectures.219 220**Q: Does ZenDNN support quantized models?**221 222A: Yes. The ZenDNN backend supports Q8_0 quantized models for supported matrix multiplication operations. FP32 and BF16 are also supported. Other quantization formats may fall back to the standard CPU backend unless explicitly supported by the ZenDNN backend.223 224**Q: Why is my inference not faster with ZenDNN?**225 226A: Ensure:2271. You're using an AMD EPYC or Ryzen processor (Zen 2 or newer)2282. `ZENDNNL_MATMUL_ALGO=1` is set for best performance (Blocked AOCL DLP)2293. You're using a sufficiently large model (small models may not benefit as much)2304. Enable profiling to verify ZenDNN MatMul is being called231 232### **GitHub Contribution**:233Please add the **[ZenDNN]** prefix/tag in issues/PRs titles to help the ZenDNN-team check/address them without delay.234 235## TODO236 237- Expand operation support beyond MUL_MAT and MUL_MAT_ID (attention operations, activations, etc.)238