Felipe97/llama-cpp-compiled
01.1k
1# llama.cpp for CANN2 3 - [Background](#background)4 - [News](#news)5 - [OS](#os)6 - [Hardware](#hardware)7 - [Model Supports](#model-supports)8 - [DataType Supports](#datatype-supports)9 - [Docker](#docker)10 - [Linux](#linux)11 - [Environment variable setup](#environment-variable-setup)12 - [TODO](#todo)13 14 15## Background16 17**Ascend NPU** is a range of AI processors using Neural Processing Unit. It will efficiently handle matrix-matrix multiplication, dot-product and scalars.18 19**CANN** (Compute Architecture for Neural Networks) is a heterogeneous computing architecture for AI scenarios, providing support for multiple AI frameworks on the top and serving AI processors and programming at the bottom. It plays a crucial role in bridging the gap between upper and lower layers, and is a key platform for improving the computing efficiency of Ascend AI processors. Meanwhile, it offers a highly efficient and easy-to-use programming interface for diverse application scenarios, allowing users to rapidly build AI applications and services based on the Ascend platform.20 21**Llama.cpp + CANN**22 23The llama.cpp CANN backend is designed to support Ascend NPU. It utilize the ability of AscendC and ACLNN which are integrated to CANN Toolkit and kernels to using Ascend NPU directly.24 25## News26 27- 2024.1128 - Support F16 and F32 data type model for Ascend 310P NPU.29- 2024.830 - Support `Q4_0` and `Q8_0` data type for Ascend NPU.31- 2024.732 - Create CANN backend for Ascend NPU.33 34## OS35 36| OS | Status | Verified |37|:-------:|:-------:|:----------------------------------------------:|38| Linux | Support | Ubuntu 22.04, OpenEuler22.03 |39 40 41## Hardware42 43### Ascend NPU44 45You can retrieve your Ascend device IDs using the following command:46 47```sh48lspci -n | grep -Eo '19e5:d[0-9a-f]{3}' | cut -d: -f249```50 51**Devices**52 53| Device Id | Product Series | Product Models | Chip Model | Verified Status |54|:---------:|----------------|----------------|:----------:|:---------------:|55| d803 | Atlas A3 Train | | 910C | |56| d803 | Atlas A3 Infer | | 910C | |57| d802 | Atlas A2 Train | | 910B | |58| d802 | Atlas A2 Infer | Atlas 300I A2 | 910B | Support |59| d801 | Atlas Train | | 910 | |60| d500 | Atlas Infer | Atlas 300I Duo | 310P | Support |61 62*Notes:*63 64- If you have trouble with Ascend NPU device, please create a issue with **[CANN]** prefix/tag.65- If you run successfully with your Ascend NPU device, please help update the upper table.66 67 68## Model Supports69 70<details>71<summary>Text-only</summary>72 73| Model Name | FP16 | Q4_0 | Q8_0 |74|:----------------------------|:-----:|:----:|:----:|75| Llama-2 | √ | √ | √ |76| Llama-3 | √ | √ | √ |77| Mistral-7B | √ | √ | √ |78| Mistral MOE | √ | √ | √ |79| DBRX | - | - | - |80| Falcon | √ | √ | √ |81| Chinese LLaMA/Alpaca | √ | √ | √ |82| Vigogne(French) | √ | √ | √ |83| BERT | x | x | x |84| Koala | √ | √ | √ |85| Baichuan | √ | √ | √ |86| Aquila 1 & 2 | √ | √ | √ |87| Starcoder models | √ | √ | √ |88| Refact | √ | √ | √ |89| MPT | √ | √ | √ |90| Bloom | √ | √ | √ |91| Yi models | √ | √ | √ |92| stablelm models | √ | √ | √ |93| DeepSeek models | x | x | x |94| Qwen models | √ | √ | √ |95| PLaMo-13B | √ | √ | √ |96| Phi models | √ | √ | √ |97| PhiMoE | √ | √ | √ |98| GPT-2 | √ | √ | √ |99| Orion | √ | √ | √ |100| InternlLM2 | √ | √ | √ |101| CodeShell | √ | √ | √ |102| Gemma | √ | √ | √ |103| Mamba | √ | √ | √ |104| Xverse | √ | √ | √ |105| command-r models | √ | √ | √ |106| Grok-1 | - | - | - |107| SEA-LION | √ | √ | √ |108| GritLM-7B | √ | √ | √ |109| OLMo | √ | √ | √ |110| OLMo 2 | √ | √ | √ |111| OLMoE | √ | √ | √ |112| Granite models | √ | √ | √ |113| GPT-NeoX | √ | √ | √ |114| Pythia | √ | √ | √ |115| Snowflake-Arctic MoE | - | - | - |116| Smaug | √ | √ | √ |117| Poro 34B | √ | √ | √ |118| Bitnet b1.58 models | √ | x | x |119| Flan-T5 | √ | √ | √ |120| Open Elm models | x | √ | √ |121| chatGLM3-6B + ChatGLM4-9b + GLMEdge-1.5b + GLMEdge-4b | √ | √ | √ |122| GLM-4-0414 | √ | √ | √ |123| SmolLM | √ | √ | √ |124| EXAONE-3.0-7.8B-Instruct | √ | √ | √ |125| FalconMamba Models | √ | √ | √ |126| Jais Models | - | x | x |127| Bielik-11B-v2.3 | √ | √ | √ |128| RWKV-6 | - | √ | √ |129| QRWKV-6 | √ | √ | √ |130| GigaChat-20B-A3B | x | x | x |131| Trillion-7B-preview | √ | √ | √ |132| Ling models | √ | √ | √ |133 134</details>135 136<details>137<summary>Multimodal</summary>138 139| Model Name | FP16 | Q4_0 | Q8_0 |140|:----------------------------|:-----:|:----:|:----:|141| LLaVA 1.5 models, LLaVA 1.6 models | x | x | x |142| BakLLaVA | √ | √ | √ |143| Obsidian | √ | - | - |144| ShareGPT4V | x | - | - |145| MobileVLM 1.7B/3B models | - | - | - |146| Yi-VL | - | - | - |147| Mini CPM | √ | √ | √ |148| Moondream | √ | √ | √ |149| Bunny | √ | - | - |150| GLM-EDGE | √ | √ | √ |151| Qwen2-VL | √ | √ | √ |152 153</details>154 155 156 157## DataType Supports158 159| DataType | 910B | 310P |160|:----------------------:|:-------:|:-------:|161| FP16 | Support | Support |162| Q8_0 | Support | Partial |163| Q4_0 | Support | Partial |164| BF16 | Support | |165 166> **310P note**167> - `Q8_0`: data transform / buffer path is implemented, and `GET_ROWS` is supported, but quantized `MUL_MAT` / `MUL_MAT_ID` are not supported.168> - `Q4_0`: data transform / buffer path is implemented, but quantized `MUL_MAT` / `MUL_MAT_ID` are not supported.169 170## Docker171 172### Build Images173You can get a image with llama.cpp in one command.174```sh175docker build -t llama-cpp-cann -f .devops/llama-cli-cann.Dockerfile .176```177 178### Run container179 180```sh181# Find all cards.182npu-smi info183 184# Select the cards that you want to use, make sure these cards are not used by someone.185# Following using cards of device0.186docker run --name llamacpp \187 --device /dev/davinci0 \188 --device /dev/davinci_manager \189 --device /dev/devmm_svm \190 --device /dev/hisi_hdc \191 -v /usr/local/dcmi:/usr/local/dcmi \192 -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \193 -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \194 -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \195 -v /PATH_TO_YOUR_MODELS/:/app/models \196 -it llama-cpp-cann \197 -m /app/models/MODEL_PATH \198 -ngl 32 \199 -p "Building a website can be done in 10 simple steps:"200```201 202*Notes:*203 204- You may need to install Ascend Driver and firmware on the **host** machine *(Please refer to the [Linux configuration](#linux) for details)*.205 206## Linux207 208### I. Setup Environment209 2101. **Configure Ascend user and group**211 212 ```sh213 sudo groupadd HwHiAiUser214 sudo useradd -g HwHiAiUser -d /home/HwHiAiUser -m HwHiAiUser -s /bin/bash215 sudo usermod -aG HwHiAiUser $USER216 ```217 2182. **Install dependencies**219 220 **Ubuntu/Debian:**221 ```sh222 sudo apt-get update223 sudo apt-get install -y gcc python3 python3-pip linux-headers-$(uname -r)224 ```225 226 **RHEL/CentOS:**227 ```sh228 sudo yum makecache229 sudo yum install -y gcc python3 python3-pip kernel-headers-$(uname -r) kernel-devel-$(uname -r)230 ```231 2323. **Install CANN (driver + toolkit)**233 234 > The `Ascend-cann` package includes both the driver and toolkit.235 > `$ARCH` can be `x86_64` or `aarch64`, `$CHIP` can be `910b` or `310p`.236 237 ```sh238 wget https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/CANN/CANN%208.5.T63/Ascend-cann_8.5.0_linux-$ARCH.run239 sudo bash ./Ascend-cann_8.5.0_linux-$ARCH.run --install240 241 wget https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/CANN/CANN%208.5.T63/Ascend-cann-$CHIP-ops_8.5.0_linux-$ARCH.run242 sudo bash ./Ascend-cann-$CHIP-ops_8.5.0_linux-$ARCH.run --install243 ```244 2454. **Verify installation**246 247 ```sh248 npu-smi info249 ```250 251 If device information is displayed correctly, the driver is functioning properly.252 253 ```sh254 # Set environment variables (adjust path if needed)255 source /usr/local/Ascend/cann/set_env.sh256 257 python3 -c "import acl; print(acl.get_soc_name())"258 ```259 260 If the command outputs the chip model, the installation was successful.261 262### II. Build llama.cpp263 264```sh265cmake -B build -DGGML_CANN=on -DCMAKE_BUILD_TYPE=release266cmake --build build --config release267```268 269### III. Run the inference270 2711. **Retrieve and prepare model**272 273 You can refer to the general [*Obtaining and quantizing models*](../../README.md#obtaining-and-quantizing-models) guide for model prepration.274 275 **Notes**:276 277 - CANN backend only supports FP16/Q4_0/Q8_0 models currently.278 2792. **Launch inference**280 281 There are two device selection modes:282 283 - Single device: Use one device target specified by the user.284 - Multiple devices: Automatically choose the devices with the same backend.285 286 | Device selection | Parameter |287 |:----------------:|:--------------------------------------:|288 | Single device | --split-mode none --main-gpu DEVICE_ID |289 | Multiple devices | --split-mode layer (default) |290 291 Examples:292 293 - Use device 0:294 295 ```sh296 ./build/bin/llama-cli -m path_to_model -p "Building a website can be done in 10 simple steps:" -n 400 -e -ngl 33 -sm none -mg 0297 ```298 299 - Use multiple devices:300 301 ```sh302 ./build/bin/llama-cli -m path_to_model -p "Building a website can be done in 10 simple steps:" -n 400 -e -ngl 33 -sm layer303 ```304 305### **GitHub contribution**:306Please add the **[CANN]** prefix/tag in issues/PRs titles to help the CANN-team check/address them without delay.307 308## Updates309### Basic Flash Attention Support310The basic FA kernel with aclnnops has been added in aclnn_ops.cpp.311Currently, the FA only supports the cases with FP16 KV tensors and NO logit softcap.312Since the aclnn interface for flash attention cannot support the logit softcap, we will only update the quantized version in the future.313 314Authors from Peking University: Bizhao Shi (bshi@pku.edu.cn), Yuxin Yang (yxyang@pku.edu.cn), Ruiyang Ma (ruiyang@stu.pku.edu.cn), and Guojie Luo (gluo@pku.edu.cn).315 316We would like to thank Tuo Dai, Shanni Li, and all of the project maintainers from Huawei Technologies Co., Ltd for their help during the code development and pull request.317 318## Environment variable setup319 320### GGML_CANN_MEM_POOL321 322Specifies the memory pool management strategy, Default is vmm.323 324- vmm: Utilizes a virtual memory manager pool. If hardware support for VMM is unavailable, falls back to the legacy (leg) memory pool.325 326- prio: Employs a priority queue-based memory pool management.327 328- leg: Uses a fixed-size buffer pool.329 330### GGML_CANN_DISABLE_BUF_POOL_CLEAN331 332Controls automatic cleanup of the memory pool. This option is only effective when using the prio or leg memory pool strategies.333 334### GGML_CANN_WEIGHT_NZ335 336Converting the matmul weight format from ND to NZ to improve performance. Enabled by default.337 338### GGML_CANN_ACL_GRAPH339 340Operators are executed using ACL graph execution, rather than in op-by-op (eager) mode. Enabled by default. This option is only effective if `USE_ACL_GRAPH` was enabled at compilation time. To enable it, recompile using:341 342```sh343cmake -B build -DGGML_CANN=on -DCMAKE_BUILD_TYPE=release -DUSE_ACL_GRAPH=ON344cmake --build build --config release345```346 347### GGML_CANN_GRAPH_CACHE_CAPACITY348 349Maximum number of compiled CANN graphs kept in the LRU cache, default is 12. When the number of cached graphs exceeds this capacity, the least recently used graph will be evicted.350 351### GGML_CANN_PREFILL_USE_GRAPH352 353Enable ACL graph execution during the prefill stage, default is false. This option is only effective when FA is enabled.354 355### GGML_CANN_OPERATOR_FUSION356 357Enable operator fusion during computation, default is false. This option fuses compatible operators (e.g., ADD + RMS_NORM) to reduce overhead and improve performance.358