Felipe97/llama-cpp-compiled
01.1k
1# OpenVINO Backend for llama.cpp2 3> [!NOTE]4> Performance and memory optimizations, accuracy validation, broader quantization coverage, broader operator and model support are work in progress.5 6[OpenVINO](https://docs.openvino.ai/) is an open-source toolkit for optimizing and deploying high-performance AI inference, specifically designed for Intel hardware, including CPUs, GPUs, and NPUs, in the cloud, on-premises, and on the edge. [OpenVINO backend for llama.cpp](../../ggml/src/ggml-openvino) enables hardware-accelerated inference on **Intel® CPUs, GPUs, and NPUs** while remaining compatible with the existing **GGUF model ecosystem**. The backend translates GGML compute graphs into OpenVINO graphs and leverages graph compilation, kernel fusion, and device-specific optimizations to improve inference performance on supported Intel hardware.7 8The OpenVINO backend is implemented in `ggml/src/ggml-openvino` and provides a translation layer for core GGML operations. The OpenVINO backend replaces the standard GGML graph execution path with Intel's OpenVINO inference engine. This approach allows the same GGUF model file to run on Intel CPUs, Intel GPUs (integrated and discrete), and Intel NPUs without changes to the model or the rest of the llama.cpp stack. When a `ggml_cgraph` is dispatched to OpenVINO backend, it:9 10- Walks the GGML graph and identifies inputs, outputs, weights, and KV cache tensors.11- Translates the GGML operations into an `ov::Model` using OpenVINO's frontend API.12- Compiles and caches the model for the target device.13- Binds GGML tensor memory to OpenVINO inference tensors and runs inference.14 15For guidance on contributing to the OpenVINO backend, see the [OpenVINO Backend Contributing Guide](https://github.com/ravi9/llamacpp-ov-dev-guide/blob/main/contributing-llamacpp-ov.md).16 17## Contents18 19- [Supported Devices](#supported-devices)20- [Supported Model Precisions](#supported-model-precisions)21- [Supported Llama.cpp Tools](#supported-llamacpp-tools)22- [Validated Models](#validated-models)23- [Build Instructions](#build-instructions)24 - [0. Prerequisites](#0-prerequisites)25 - [1. Install OpenVINO Runtime](#1-install-openvino-runtime)26 - [2. Build llama.cpp with OpenVINO Backend](#2-build-llamacpp-with-openvino-backend)27 - [Ubuntu Build Script](#ubuntu-build-script)28 - [Windows Build Script](#windows-build-script)29 - [3. Download Sample Model](#3-download-sample-model)30 - [4. Run Inference with OpenVINO Backend](#4-run-inference-with-openvino-backend)31 - [5. Docker Build](#5-docker-build)32- [GGML OpenVINO Backend Runtime Configurations](#ggml-openvino-backend-runtime-configurations)33- [Known Limitations](#known-limitations)34- [Work in Progress](#work-in-progress)35 36## Supported Devices37 38OpenVINO backend supports the following hardware:39 40- Intel CPUs41- Intel GPUs (integrated and discrete)42- Intel NPUs43 44Although OpenVINO supports a wide range of [Intel hardware](https://docs.openvino.ai/2026/about-openvino/release-notes-openvino/system-requirements.html), the llama.cpp OpenVINO backend has been validated specifically on AI PCs such as the Intel® Core™ Ultra Series 1 and Series 2.45 46## Supported Model Precisions47 48- `FP16`49- `BF16` (on Intel Xeon)50- `Q8_0`51- `Q4_0`52- `Q4_1`53- `Q4_K`54- `Q4_K_M`55- `Q5_K` (converted to `Q8_0_C` at runtime)56- `Q6_K` (converted to `Q8_0_C` at runtime)57 58> [!NOTE]59> Accuracy validation and performance optimizations for quantized models are a work in progress.60 61**CPU and GPU Quantization Details:**62- `Q5_K` and `Q6_K` tensors are converted to `Q8_0_C`63 64**NPU Quantization Details:**65- Primary supported quantization scheme is `Q4_0`66- `Q6_K` tensors are requantized to `Q4_0_128` in general. For embedding weights, `Q6_K` tensors are requantized to `Q8_0_C` except for the token embedding matrix which is dequantized to fp1667 68**Additional Notes:**69- Both `Q4_0` and `Q4_1` models use `Q6_K` for the token embedding tensor and the final matmul weight tensor (often the same tensor)70- `Q4_0` models may produce some `Q4_1` tensors if an imatrix is provided during quantization using `llama-quantize`71- `Q4_K_M` models may include both `Q6_K` and `Q5_K` tensors (observed in Phi-3)72- `Q5_1` tensors are dequantized natively (weights, scales, and zero-points extracted directly)73 74## Supported Llama.cpp Tools75 76The OpenVINO backend integrates with the standard llama.cpp tools listed below.77However, all the tools coverage across all devices is not uniform and exhaustive validation is work in progress.78 79- llama-bench80- llama-cli81- llama-completion82- llama-embedding83- llama-perplexity84- llama-run85- llama-server86- llama-simple87 88## Validated Models89 90Although, the validated models below were tested with `llama-cli` using the `Q4_K_M` quantization format on Intel® Core™ Ultra Series 2 (Lunar Lake), the OpenVINO backend is expected to work across a broader range of [Intel hardware](https://docs.openvino.ai/2026/about-openvino/release-notes-openvino/system-requirements.html), [supported model precisions](#supported-model-precisions), [supported llama.cpp tools](#supported-llamacpp-tools) and additional model architectures.91 92> [!NOTE]93> Extensive accuracy validation, performance optimizations, and broader architecture coverage are work in progress.94 95**Legend & Test Configuration:**96- **Status:** ✓ = Passed | ✗ = Failed or Unsupported97- **Execution Modes:**98 - **SL** = Stateless (`GGML_OPENVINO_STATEFUL_EXECUTION=0`)99 - **SF** = Stateful (`GGML_OPENVINO_STATEFUL_EXECUTION=1`)100 - Note: The NPU operates in stateless mode only.101- **Validation system:** Intel® Core™ Ultra 5 238V (Lunar Lake) | 32 GB RAM | Ubuntu 24.04 | Intel Graphics Compiler 2.41.5 | Intel OpenCL GPU Driver 26.31.39395.13-0 | Intel NPU Driver 1.38.0.102- See [Known Limitations](#known-limitations) for context on observed failures.103 104| Model | CPU (SL / SF) | GPU (SL / SF) | NPU (SL) |105| :--- | :---: | :---: | :---: |106| [bartowski/Llama-3.2-1B-Instruct-Q4_K_M](https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |107| [bartowski/Llama-3.2-3B-Instruct-Q4_K_M](https://huggingface.co/bartowski/Llama-3.2-3B-Instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |108| [bartowski/Meta-Llama-3.1-8B-Instruct-Q4_K_M](https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |109| | | | |110| [Qwen/qwen2.5-1.5b-instruct-q4_k_m](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |111| [Qwen/qwen2.5-coder-7b-instruct-q4_k_m](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |112| [bartowski/Qwen_Qwen3-0.6B-Q4_K_M](https://huggingface.co/bartowski/Qwen_Qwen3-0.6B-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |113| [bartowski/Qwen_Qwen3-1.7B-Q4_K_M](https://huggingface.co/bartowski/Qwen_Qwen3-1.7B-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |114| [Qwen/Qwen3-4B-Q4_K_M](https://huggingface.co/Qwen/Qwen3-4B-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |115| [lm-kit/Qwen3-8B-Q4_K_M](https://huggingface.co/lm-kit/qwen-3-8b-instruct-gguf) | ✓ / ✓ | ✓ / ✓ | ✓ |116| [bartowski/Qwen_Qwen3.5-0.8B-Q4_K_M](https://huggingface.co/bartowski/Qwen_Qwen3.5-0.8B-GGUF) | ✓ / ✗ | ✓ / ✗ | ✗ |117| [bartowski/Qwen_Qwen3.5-2B-Q4_K_M](https://huggingface.co/bartowski/Qwen_Qwen3.5-2B-GGUF) | ✓ / ✗ | ✓ / ✗ | ✗ |118| [bartowski/Qwen_Qwen3.5-4B-Q4_K_M](https://huggingface.co/bartowski/Qwen_Qwen3.5-4B-GGUF) | ✓ / ✗ | ✓ / ✗ | ✗ |119| [lmstudio-community/Qwen3.5-9B-Q4_K_M](https://huggingface.co/lmstudio-community/Qwen3.5-9B-GGUF) | ✓ / ✗ | ✓ / ✗ | ✗ |120| | | | |121| [unsloth/gemma-3-4b-it-Q4_K_M](https://huggingface.co/unsloth/gemma-3-4b-it-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |122| [bartowski/google_gemma-4-E2B-it-Q4_K_M](https://huggingface.co/bartowski/google_gemma-4-E2B-it-GGUF) | ✓ / ✓ | ✓ / ✓ | ✗ |123| [bartowski/google_gemma-4-E4B-it-Q4_K_M](https://huggingface.co/bartowski/google_gemma-4-E4B-it-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |124| [bartowski/gemma-4-12B-it-Q4_K_M](https://huggingface.co/bartowski/gemma-4-12B-it-GGUF) | ✓ / ✓ | ✓ / ✓ | ✗ |125| | | | |126| [bartowski/Phi-3-mini-4k-instruct-Q4_K_M](https://huggingface.co/bartowski/Phi-3-mini-4k-instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |127| [bartowski/Phi-3.5-mini-instruct-Q4_K_M](https://huggingface.co/bartowski/Phi-3.5-mini-instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |128| [bartowski/microsoft_Phi-4-mini-instruct-Q4_K_M](https://huggingface.co/bartowski/microsoft_Phi-4-mini-instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |129| | | | |130| [bartowski/Mistral-7B-Instruct-v0.3-Q4_K_M](https://huggingface.co/bartowski/Mistral-7B-Instruct-v0.3-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |131| [QuantFactory/Ministral-3b-instruct.Q4_K_M](https://huggingface.co/QuantFactory/Ministral-3b-instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |132| [bartowski/Ministral-8B-Instruct-2410-Q4_K_M](https://huggingface.co/bartowski/Ministral-8B-Instruct-2410-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |133| | | | |134| [bartowski/DeepSeek-R1-Distill-Llama-8B-Q4_K_M](https://huggingface.co/bartowski/DeepSeek-R1-Distill-Llama-8B-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |135| [bartowski/DeepSeek-R1-Distill-Qwen-7B-Q4_K_M](https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-7B-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |136| | | | |137| [ibm-granite/granite-4.0-350m-Q4_K_M](https://huggingface.co/ibm-granite/granite-4.0-350m-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |138| [ibm-granite/granite-4.0-micro-Q4_K_M](https://huggingface.co/ibm-granite/granite-4.0-micro-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |139| [ibm-granite/granite-4.0-1b-Q4_K_M](https://huggingface.co/ibm-granite/granite-4.0-1b-GGUF) | ✓ / ✓ | ✓ / ✓ | ✗ |140| [ibm-research/granite-3.2-8b-instruct-Q4_K_M](https://huggingface.co/ibm-research/granite-3.2-8b-instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |141| | | | |142| [HuggingFaceTB/smollm2-1.7b-instruct-q4_k_m](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |143| [openbmb/MiniCPM-V-2_6-Q4_K_M](https://huggingface.co/openbmb/MiniCPM-V-2_6-gguf) | ✓ / ✓ | ✓ / ✓ | ✓ |144| [bartowski/tencent_Hunyuan-7B-Instruct-Q4_K_M](https://huggingface.co/bartowski/tencent_Hunyuan-7B-Instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |145| [LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct-Q4_K_M](https://huggingface.co/LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |146| [bartowski/prism-ml_Bonsai-8B-unpacked-Q4_K_M](https://huggingface.co/bartowski/prism-ml_Bonsai-8B-unpacked-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |147| | | | |148| [gpustack/bge-m3-Q4_K_M.gguf](https://huggingface.co/gpustack/bge-m3-GGUF) | ✓ | ✗ | ✗ |149 150## Build Instructions151 152### 0. Prerequisites153 154- Linux or Windows system with Intel hardware (CPU, GPU, or NPU)155- **For Intel GPU or NPU Usage**: Install the appropriate hardware drivers for your Intel GPU or NPU. For detailed instructions, see: [Additional Configurations for Hardware Acceleration](https://docs.openvino.ai/2026/get-started/install-openvino/configurations.html).156 157- **Linux:**158 - Git, CMake, and Ninja software tools are needed for building.159 ```bash160 sudo apt-get update161 sudo apt-get install -y build-essential libcurl4-openssl-dev libtbb12 cmake ninja-build python3-pip curl wget tar162 ```163 - OpenCL164 ```bash165 sudo apt install ocl-icd-opencl-dev opencl-headers opencl-clhpp-headers intel-opencl-icd166 ```167 168- **Windows:**169 - Download and install [Microsoft Visual Studio 2022 Build Tools](https://aka.ms/vs/17/release/vs_BuildTools.exe). During installation, select the **"Desktop development with C++"** workload.170 171 - Install required tools:172 ```powershell173 # Windows PowerShell174 winget install Git.Git175 winget install GNU.Wget176 winget install Ninja-build.Ninja177 ```178 179 - Install **OpenCL** using **vcpkg**:180 ```powershell181 # Windows PowerShell182 cd C:\183 git clone https://github.com/microsoft/vcpkg184 cd vcpkg185 .\bootstrap-vcpkg.bat186 .\vcpkg install opencl187 # Optional but recommended: Integrate vcpkg with Visual Studio / CMake:188 .\vcpkg integrate install189 ```190 191### 1. Install OpenVINO Runtime192 193- Follow the guide to install OpenVINO Runtime from an archive file: [Linux](https://docs.openvino.ai/2026/get-started/install-openvino/install-openvino-archive-linux.html) | [Windows](https://docs.openvino.ai/2026/get-started/install-openvino/install-openvino-archive-windows.html)194 195- Verify OpenVINO is initialized properly:196 ```bash197 echo $OpenVINO_DIR198 ```199 200### 2. Build llama.cpp with OpenVINO Backend201 202Clone llama.cpp repo and build :203 204```bash205git clone https://github.com/ggml-org/llama.cpp206cd llama.cpp207```208 209- **Linux:**210```bash211source /opt/intel/openvino/setupvars.sh212cmake -B build/ReleaseOV -G Ninja -DCMAKE_BUILD_TYPE=Release -DGGML_OPENVINO=ON213cmake --build build/ReleaseOV --parallel214```215 216- **Windows:** Open **x64 Native Tools Command Prompt for VS** (so the MSVC toolchain is on `PATH`), then run:217 218```cmd219C:\Intel\openvino\setupvars.bat220cmake -B build\ReleaseOV -G Ninja -DCMAKE_BUILD_TYPE=Release -DGGML_OPENVINO=ON -DCMAKE_TOOLCHAIN_FILE=C:\vcpkg\scripts\buildsystems\vcpkg.cmake221cmake --build build\ReleaseOV --parallel222```223 224> [!NOTE]225> The Windows install path is `C:\Intel\openvino` (no spaces) to avoid quoting problems some CMake/Ninja toolchains have with `C:\Program Files (x86)\...`. Adjust to wherever you installed OpenVINO Runtime. From `cmd`, run `C:\Intel\openvino\setupvars.bat`; from PowerShell, run `& "C:\Intel\openvino\setupvars.ps1"` instead. Once the build is finished you can launch the binaries from any `cmd` or `PowerShell` window after sourcing the matching `setupvars` script for that shell.226 227#### Ubuntu Build Script228 229For Ubuntu24 users, the following shell script automates the prerequisite installs (build tools, OpenCL ICD), the OpenVINO Runtime download/extract/setup, and the Ninja-based llama.cpp build.230Save the following as `build-llamacpp-ov.sh` next to where you want the `llama.cpp` folder to land, then run it:231 232```bash233chmod +x build-llamacpp-ov.sh234./build-llamacpp-ov.sh235```236 237<details>238<summary>Click to expand <code>build-llamacpp-ov.sh</code></summary>239 240```bash241#!/usr/bin/env bash242# ============================================243# llama.cpp OpenVINO Build Script (Ninja)244# ============================================245set -euo pipefail246 247OPENVINO_VERSION_MAJOR="2026.4"248OPENVINO_VERSION_FULL="2026.4.0.22959.99c81491cc3"249 250SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"251OPENVINO_INSTALL_DIR="/opt/intel/openvino_${OPENVINO_VERSION_MAJOR}"252OPENVINO_LINK_DIR="/opt/intel/openvino"253OPENVINO_TGZ="${SCRIPT_DIR}/openvino.tgz"254OPENVINO_URL="https://storage.openvinotoolkit.org/repositories/openvino/packages/${OPENVINO_VERSION_MAJOR}/linux/openvino_toolkit_ubuntu24_${OPENVINO_VERSION_FULL}_x86_64.tgz"255 256echo "============================================"257echo "Installing prerequisites (apt)..."258echo "============================================"259sudo apt-get update260sudo apt-get install -y \261 build-essential libcurl4-openssl-dev libtbb12 \262 cmake ninja-build python3-pip \263 curl wget tar git264 265echo "============================================"266echo "Installing OpenCL runtime + headers..."267echo "============================================"268sudo apt-get install -y \269 ocl-icd-opencl-dev opencl-headers opencl-clhpp-headers intel-opencl-icd270 271cd "${SCRIPT_DIR}"272 273# ============================================274# Clone llama.cpp if missing275# ============================================276if [[ ! -f "llama.cpp/CMakeLists.txt" ]]; then277 echo "Cloning llama.cpp..."278 git clone https://github.com/ggml-org/llama.cpp279fi280 281# ============================================282# Setup OpenVINO: download & extract to /opt/intel/openvino_${OPENVINO_VERSION_MAJOR},283# then point /opt/intel/openvino at it via symlink so the active version is swappable.284# ============================================285if [[ -f "${OPENVINO_INSTALL_DIR}/setupvars.sh" ]]; then286 echo "OpenVINO ${OPENVINO_VERSION_MAJOR} already installed at ${OPENVINO_INSTALL_DIR}. Skipping download."287else288 echo "OpenVINO not found at ${OPENVINO_INSTALL_DIR}. Starting download..."289 curl -L -o "${OPENVINO_TGZ}" "${OPENVINO_URL}"290 291 echo "Extracting OpenVINO to ${OPENVINO_INSTALL_DIR}..."292 sudo mkdir -p "${OPENVINO_INSTALL_DIR}"293 sudo tar -xzf "${OPENVINO_TGZ}" -C "${OPENVINO_INSTALL_DIR}" --strip-components=1294 rm -f "${OPENVINO_TGZ}"295fi296 297# Refresh symlink: /opt/intel/openvino -> /opt/intel/openvino_${OPENVINO_VERSION_MAJOR}298sudo ln -sfn "${OPENVINO_INSTALL_DIR}" "${OPENVINO_LINK_DIR}"299 300OPENVINO_ROOT="${OPENVINO_LINK_DIR}"301echo "OpenVINO Ready: ${OPENVINO_ROOT} -> ${OPENVINO_INSTALL_DIR}"302 303# Install OpenVINO's own runtime dependencies (one-time per system).304if [[ -x "${OPENVINO_ROOT}/install_dependencies/install_openvino_dependencies.sh" ]]; then305 echo "============================================"306 echo "Installing OpenVINO runtime dependencies..."307 echo "============================================"308 echo "Y" | sudo -E "${OPENVINO_ROOT}/install_dependencies/install_openvino_dependencies.sh"309fi310 311# ============================================312# Clean old build cache313# ============================================314cd "${SCRIPT_DIR}/llama.cpp"315if [[ -d "build/ReleaseOV" ]]; then316 echo "Removing old build directory..."317 rm -rf "build/ReleaseOV"318fi319 320echo "============================================"321echo "Configuring with CMake..."322echo "============================================"323set +u324source "${OPENVINO_ROOT}/setupvars.sh"325set -u326 327cmake -B build/ReleaseOV -G Ninja \328 -DCMAKE_BUILD_TYPE=Release \329 -DGGML_OPENVINO=ON330 331cmake --build build/ReleaseOV --parallel332 333echo "============================================"334echo "Build completed successfully!"335echo "============================================"336echo "Binaries: $(pwd)/build/ReleaseOV/bin"337echo338echo "NOTE: To run, source setupvars.sh and pick a device:"339echo " source /opt/intel/openvino/setupvars.sh"340echo " export GGML_OPENVINO_DEVICE=CPU # or GPU / NPU"341echo " ./build/ReleaseOV/bin/llama-cli -m model.gguf"342```343 344> [!NOTE]345> The script pins OpenVINO `2026.4` via the `OPENVINO_VERSION_MAJOR` / `OPENVINO_VERSION_FULL` variables at the top — edit them to track a different release.346 347</details>348 349#### Windows Build Script350 351For Windows users, the following `.bat` script automates the prerequisite installs (Git, Ninja, CMake, Visual Studio 2022 Build Tools, vcpkg + OpenCL), the OpenVINO Runtime download/extract, and the Ninja-based llama.cpp build.352Save the following as `build-llamacpp-ov.bat` next to where you want the `llama.cpp` to land, then run it from either **Command Prompt** or **PowerShell**:353 354```cmd355:: Command Prompt356build-llamacpp-ov.bat357```358 359```powershell360# PowerShell361.\build-llamacpp-ov.bat362```363 364<details>365<summary>Click to expand <code>build-llamacpp-ov.bat</code></summary>366 367```bat368@echo off369setlocal enabledelayedexpansion370 371REM ============================================372REM llama.cpp OpenVINO Build Script (Ninja)373REM ============================================374 375set "OPENVINO_VERSION_MAJOR=2026.4"376set "OPENVINO_VERSION_FULL=2026.4.0.22959.99c81491cc3"377 378set "SCRIPT_DIR=%~dp0"379set "VCPKG_DIR=C:\vcpkg"380set "OPENVINO_INSTALL_DIR=C:\Intel\openvino_%OPENVINO_VERSION_MAJOR%"381set "OPENVINO_LINK_DIR=C:\Intel\openvino"382set "OPENVINO_ZIP=%SCRIPT_DIR%openvino.zip"383set "OPENVINO_EXTRACT_TMP=%SCRIPT_DIR%openvino_extract_tmp"384set "OPENVINO_URL=https://storage.openvinotoolkit.org/repositories/openvino/packages/%OPENVINO_VERSION_MAJOR%/windows/openvino_toolkit_windows_%OPENVINO_VERSION_FULL%_x86_64.zip"385 386echo ============================================387echo Installing prerequisites...388echo ============================================389winget install --id Git.Git -e --accept-source-agreements --accept-package-agreements 2>nul390winget install --id Ninja-build.Ninja -e --accept-source-agreements --accept-package-agreements 2>nul391winget install --id Kitware.CMake -e --accept-source-agreements --accept-package-agreements 2>nul392 393REM Ensure Visual Studio Build Tools are installed.394echo Checking for Visual Studio Build Tools...395set "VSWHERE=%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe"396set "VS_INSTALLED="397if exist "%VSWHERE%" (398 for /f "usebackq tokens=*" %%i in (`"%VSWHERE%" -latest -products * -requires Microsoft.VisualStudio.Component.VC.Tools.x86.x64 -property installationPath 2^>nul`) do (399 set "VS_INSTALLED=%%i"400 )401)402if defined VS_INSTALLED (403 echo Visual Studio with VC++ x86/x64 tools already present at "!VS_INSTALLED!". Skipping winget install.404) else (405 winget install --id Microsoft.VisualStudio.2022.BuildTools -e --override "--wait --passive --add Microsoft.VisualStudio.Workload.VCTools --includeRecommended" --accept-source-agreements --accept-package-agreements406 if errorlevel 1 (407 echo WARNING: winget could not install Visual Studio Build Tools automatically.408 echo Install manually from https://aka.ms/vs/17/release/vs_BuildTools.exe ^(select the "Desktop development with C++" workload^)409 echo and re-run this script from a "Developer Command Prompt for VS 2022".410 )411)412 413echo ============================================414echo Installing OpenCL via vcpkg...415echo ============================================416if not exist "%VCPKG_DIR%" (417 git clone https://github.com/microsoft/vcpkg "%VCPKG_DIR%"418 cd /d "%VCPKG_DIR%"419 call bootstrap-vcpkg.bat420 call vcpkg integrate install421)422cd /d "%VCPKG_DIR%"423call vcpkg install opencl424 425cd /d "%SCRIPT_DIR%"426 427REM ============================================428REM Clone llama.cpp if missing429REM ============================================430if not exist "llama.cpp\CMakeLists.txt" (431 echo Cloning llama.cpp...432 git clone https://github.com/ggml-org/llama.cpp433)434 435cd /d "llama.cpp"436set "SCRIPT_DIR=%CD%"437 438REM ============================================439REM Setup OpenVINO: download & extract to C:\Intel\openvino_%OPENVINO_VERSION_MAJOR%,440REM then point C:\Intel\openvino at it via a directory junction (mklink /J).441REM ============================================442 443if exist "%OPENVINO_INSTALL_DIR%\setupvars.bat" (444 echo OpenVINO %OPENVINO_VERSION_MAJOR% already installed at "%OPENVINO_INSTALL_DIR%". Skipping download.445) else (446 echo OpenVINO not found at "%OPENVINO_INSTALL_DIR%". Starting download...447 448 curl -L -o "%OPENVINO_ZIP%" "%OPENVINO_URL%"449 if errorlevel 1 (450 echo ERROR: Download failed.451 exit /b 1452 )453 454 echo Extracting OpenVINO...455 if exist "%OPENVINO_EXTRACT_TMP%" rmdir /s /q "%OPENVINO_EXTRACT_TMP%"456 mkdir "%OPENVINO_EXTRACT_TMP%"457 tar -xf "%OPENVINO_ZIP%" -C "%OPENVINO_EXTRACT_TMP%"458 if errorlevel 1 (459 echo ERROR: Extraction failed.460 exit /b 1461 )462 463 REM Move the single top-level folder contents into the versioned install dir.464 set "OPENVINO_EXTRACTED="465 for /d %%i in ("%OPENVINO_EXTRACT_TMP%\*") do set "OPENVINO_EXTRACTED=%%i"466 if not defined OPENVINO_EXTRACTED (467 echo ERROR: Could not locate extracted OpenVINO folder under "%OPENVINO_EXTRACT_TMP%".468 exit /b 1469 )470 if not exist "%OPENVINO_INSTALL_DIR%" mkdir "%OPENVINO_INSTALL_DIR%"471 xcopy /e /i /y /q "!OPENVINO_EXTRACTED!\*" "%OPENVINO_INSTALL_DIR%\" >nul472 if errorlevel 1 (473 echo ERROR: Failed to copy OpenVINO from "!OPENVINO_EXTRACTED!" to "%OPENVINO_INSTALL_DIR%".474 echo Re-run this script from an elevated Command Prompt ^(Run as administrator^) if access is denied.475 exit /b 1476 )477 478 rmdir /s /q "%OPENVINO_EXTRACT_TMP%"479 del "%OPENVINO_ZIP%"480)481 482REM Refresh junction: C:\Intel\openvino -> C:\Intel\openvino_<version>.483REM `mklink /J` creates a directory junction (no admin / Developer Mode required).484if exist "%OPENVINO_LINK_DIR%" rmdir "%OPENVINO_LINK_DIR%"485mklink /J "%OPENVINO_LINK_DIR%" "%OPENVINO_INSTALL_DIR%" >nul486if errorlevel 1 (487 echo ERROR: Failed to create junction "%OPENVINO_LINK_DIR%" -^> "%OPENVINO_INSTALL_DIR%".488 echo If "%OPENVINO_LINK_DIR%" already exists as a regular non-empty folder, remove it manually and re-run.489 exit /b 1490)491 492set "OPENVINO_ROOT=%OPENVINO_LINK_DIR%"493echo OpenVINO Ready: %OPENVINO_ROOT% -^> %OPENVINO_INSTALL_DIR%494 495 496echo ============================================497echo Setting up compiler environment...498echo ============================================499REM Locate Visual Studio Build Tools vcvars64.bat500set "VSWHERE=%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe"501if exist "%VSWHERE%" (502 for /f "usebackq tokens=*" %%i in (`"%VSWHERE%" -latest -products Microsoft.VisualStudio.Product.BuildTools -property installationPath`) do (503 set "VS_PATH=%%i"504 )505)506if defined VS_PATH (507 call "%VS_PATH%\VC\Auxiliary\Build\vcvars64.bat" >nul508) else (509 echo WARNING: Visual Studio Build Tools not found. Compiler may be missing.510)511 512REM ============================================513REM Clean old build cache514REM ============================================515if exist "build\ReleaseOV" (516 echo Removing old build directory ...517 rmdir /s /q "build\ReleaseOV"518)519 520echo ============================================521echo Configuring with CMake...522echo ============================================523call "%OPENVINO_ROOT%\setupvars.bat" >nul 2>nul524 525cmake -B build\ReleaseOV -G Ninja ^526 -DCMAKE_BUILD_TYPE=Release ^527 -DGGML_OPENVINO=ON ^528 -DCMAKE_TOOLCHAIN_FILE="%VCPKG_DIR%\scripts\buildsystems\vcpkg.cmake"529 530if errorlevel 1 (531 echo If you continue to face CMAKE errors, make sure to install:532 echo winget install Microsoft.VisualStudio.2022.BuildTools533 echo Then run the "Developer Command Prompt for VS 2022" and launch this script from there.534 exit /b 1535)536 537cmake --build build\ReleaseOV --config Release538if errorlevel 1 exit /b 1539 540echo ============================================541echo Build completed successfully!542echo ============================================543echo Binaries: %CD%\build\ReleaseOV\bin544echo.545echo NOTE: To run, source setupvars.bat and pick a device:546echo call "C:\Intel\openvino\setupvars.bat"547echo set GGML_OPENVINO_DEVICE=CPU ^&^& REM or GPU / NPU548echo build\ReleaseOV\bin\llama-cli.exe -m model.gguf549echo.550 551endlocal552```553 554> [!NOTE]555> The script pins OpenVINO `2026.4` via the `OPENVINO_VERSION_MAJOR` / `OPENVINO_VERSION_FULL` variables at the top — edit them to track a different release. From any new shell, source the matching `setupvars` script via the junction — `call "C:\Intel\openvino\setupvars.bat"` from `cmd`, or `& "C:\Intel\openvino\setupvars.ps1"` from PowerShell. If `winget` cannot register Visual Studio Build Tools on first run, install them once manually and re-run the script from an elevated **Developer Command Prompt for VS 2022**.556 557</details>558 559 560### 3. Download Sample Model561 562Download sample model for testing.563 564```bash565# Linux566mkdir -p ~/models/567wget https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q4_K_M.gguf \568 -O ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf569 570# Windows PowerShell571mkdir C:\models572Invoke-WebRequest -Uri https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q4_K_M.gguf -OutFile C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf573 574# Windows Command Line575mkdir C:\models576curl -L https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q4_K_M.gguf -o C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf577```578 579### 4. Run Inference with OpenVINO Backend580 581When using the OpenVINO backend, the first inference token may have slightly higher latency due to on-the-fly conversion to the OpenVINO graph. Subsequent tokens and runs will be faster.582 583> [!NOTE]584> Default context size is set to the model training context, which may be very large. For example, 131072 for Llama 3.2 1B, which may result in lower performance, especially on edge/laptop devices. Use `-c` to limit context size in supported llama.cpp tools for better performance. For example, `-c 512`.585 586```bash587# If device is unset or unavailable, defaults to CPU.588# If the system has multiple GPUs, use GPU.0 or GPU.1 to explicitly target a specific GPU.589 590# Linux591export GGML_OPENVINO_DEVICE=GPU592# Optional: enable stateful execution for improved GPU performance (recommended).593export GGML_OPENVINO_STATEFUL_EXECUTION=1594# To run llama-simple:595./build/ReleaseOV/bin/llama-simple -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -n 50 "The story of AI is "596# To run in chat mode:597./build/ReleaseOV/bin/llama-cli -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -c 1024598# To run llama-bench, -fa 1 is needed599GGML_OPENVINO_STATEFUL_EXECUTION=1 GGML_OPENVINO_DEVICE=GPU ./build/ReleaseOV/bin/llama-bench -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -fa 1600 601# NPU: keep context small to avoid failures from very large model context windows.602export GGML_OPENVINO_DEVICE=NPU603./build/ReleaseOV/bin/llama-cli -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -c 512604 605# Windows Command Line606set GGML_OPENVINO_DEVICE=GPU607# Optional: enable stateful execution for improved GPU performance (recommended).608set GGML_OPENVINO_STATEFUL_EXECUTION=1609# Windows PowerShell610$env:GGML_OPENVINO_DEVICE = "GPU"611$env:GGML_OPENVINO_STATEFUL_EXECUTION = "1"612 613# To run llama-simple614build\ReleaseOV\bin\llama-simple.exe -m "C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf" -n 50 "The story of AI is "615# To run in chat mode:616build\ReleaseOV\bin\llama-cli.exe -m "C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf" -c 1024617# To run llama-bench, -fa 1 is needed618build\ReleaseOV\bin\llama-bench.exe -m "C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf" -fa 1619 620# NPU: keep context small to avoid failures from very large model context windows.621# Windows Command Line622set GGML_OPENVINO_DEVICE=NPU623# Windows PowerShell624$env:GGML_OPENVINO_DEVICE = "NPU"625build\ReleaseOV\bin\llama-cli.exe -m "C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf" -c 512626```627> [!NOTE]628> On systems with multiple GPUs, use `GPU.0` or `GPU.1` to explicitly target specific GPU. See [OpenVINO GPU Device](https://docs.openvino.ai/2026/openvino-workflow/running-inference/inference-devices-and-modes/gpu-device.html) for more details.629 630### 5. Docker Build631 632You can build and run llama.cpp with OpenVINO backend using Docker.633 634```bash635# Build the base runtime image with compiled shared libraries and minimal dependencies.636docker build -t llama-openvino:base -f .devops/openvino.Dockerfile .637 638# Build the complete image with all binaries, Python tools, gguf-py library, and model conversion utilities.639docker build --target=full -t llama-openvino:full -f .devops/openvino.Dockerfile .640 641# Build a minimal CLI-only image containing just the llama-cli executable.642docker build --target=light -t llama-openvino:light -f .devops/openvino.Dockerfile .643 644# Builds a server-only image with llama-server executable, health check endpoint, and REST API support.645docker build --target=server -t llama-openvino:server -f .devops/openvino.Dockerfile .646 647# If you are behind a proxy:648docker build --build-arg http_proxy=$http_proxy --build-arg https_proxy=$https_proxy --target=server -t llama-openvino:server -f .devops/openvino.Dockerfile .649```650 651Run llama.cpp with OpenVINO backend Docker container.652Save sample models in `~/models` as [shown above](#3-download-sample-model). It will be mounted to the container in the examples below.653 654 655```bash656# Run Docker container657docker run --rm -it -v ~/models:/models llama-openvino:light --no-warmup -c 1024 -m /models/Llama-3.2-1B-Instruct-Q4_K_M.gguf658 659# With Intel GPU access (iGPU or dGPU)660docker run --rm -it -v ~/models:/models \661--device=/dev/dri --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) -u $(id -u):$(id -g) \662--env=GGML_OPENVINO_DEVICE=GPU --env=GGML_OPENVINO_STATEFUL_EXECUTION=1 \663llama-openvino:light --no-warmup -c 1024 -m /models/Llama-3.2-1B-Instruct-Q4_K_M.gguf664 665# With Intel NPU access666docker run --rm -it -v ~/models:/models \667--device=/dev/accel --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) -u $(id -u):$(id -g) \668--env=GGML_OPENVINO_DEVICE=NPU \669llama-openvino:light --no-warmup -c 1024 -m /models/Llama-3.2-1B-Instruct-Q4_K_M.gguf670```671 672Run Llama.cpp Server with OpenVINO Backend.673> [!NOTE]674> `llama-server` with OpenVINO backend supports only one chat session/thread, when `GGML_OPENVINO_STATEFUL_EXECUTION=1` is enabled.675 676```bash677# Run the llama-openvino:server Docker container (CPU)678docker run --rm -it -p 8080:8080 -v ~/models:/models llama-openvino:server --no-warmup -m /models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -c 1024 --host 0.0.0.0679 680# Run the llama-openvino:server Docker container with Intel GPU access (iGPU or dGPU)681docker run --rm -it -v ~/models:/models \682--device=/dev/dri --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) -u $(id -u):$(id -g) \683-p 8080:8080 --env=GGML_OPENVINO_DEVICE=GPU \684llama-openvino:server --no-warmup -c 1024 -m /models/Llama-3.2-1B-Instruct-Q4_K_M.gguf --host 0.0.0.0685 686# Run the llama-openvino:server Docker container with Intel NPU access687docker run --rm -it -v ~/models:/models \688--device=/dev/accel --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) -u $(id -u):$(id -g) \689-p 8080:8080 --env=GGML_OPENVINO_DEVICE=NPU \690llama-openvino:server --no-warmup -c 1024 -m /models/Llama-3.2-1B-Instruct-Q4_K_M.gguf --host 0.0.0.0691 692# Or Using llama-server executable693./build/ReleaseOV/bin/llama-server -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf --port 8080 -c 1024694 695# Option 1: Open your browser to http://localhost:8080 to access the web UI for the llama.cpp server.696# Option 2: In a NEW terminal, test the server with curl697 698# If you are behind a proxy, make sure to set NO_PROXY to avoid proxy for localhost699export NO_PROXY=localhost,127.0.0.1700 701# Test health endpoint702curl -f http://localhost:8080/health703 704# Test with a simple prompt705curl -X POST "http://localhost:8080/v1/chat/completions" -H "Content-Type: application/json" \706 -d '{"messages":[{"role":"user","content":"Write a poem about OpenVINO"}],"max_tokens":100}' | jq .707```708 709## GGML OpenVINO Backend Runtime Configurations710 711The OpenVINO backend can be configured using the following environment variables at runtime to control device selection, caching, debugging, and profiling behavior.712Boolean flags follow a uniform convention: set to a **positive integer** (e.g. `1`) to enable; unset, empty, `0`, negative, or non-numeric values are treated as disabled.713 714| Variable | Type | Default | Description |715|-----------------------------------|-----------|------------|-------------------------------------------------------------------------------------------------------------|716| `GGML_OPENVINO_DEVICE` | String | `CPU` | Specify the target device (CPU, GPU, NPU). On systems with multiple GPUs, use `GPU.0` or `GPU.1` to explicitly target specific GPU. See [OpenVINO GPU Device](https://docs.openvino.ai/2026/openvino-workflow/running-inference/inference-devices-and-modes/gpu-device.html). When set to **NPU**, static compilation mode is enabled for optimal performance. |717| `GGML_OPENVINO_CACHE_DIR` | String | `not set` | Directory for OpenVINO model caching (recommended: `/tmp/ov_cache`). Enables model caching when set. **Not supported on NPU devices.** |718| `GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR` | String | `not set` | Directory for the frontend compiled-model cache. When set, OpenVINO compiled models are exported as blobs and imported on later runs to skip weight requantization, graph conversion, and compilation for matching single-graph models. |719| `GGML_OPENVINO_PREFILL_CHUNK_SIZE`| Integer | `256` | Token chunk size for **NPU** prefill (NPU-only; ignored on CPU/GPU). Must be a positive integer; otherwise the default is used. |720| `GGML_OPENVINO_NPU_COMPILE_CONFIG` | String | `not set` | NPU-only compiler mode parameters forwarded to OpenVINO as `NPU_COMPILATION_MODE_PARAMS`, for example `optimization-level=3`. |721| `GGML_OPENVINO_STATEFUL_EXECUTION`| Boolean | `0` | Enable stateful KV cache for better performance. Recommended on CPU, GPU. |722| `GGML_OPENVINO_DISABLE_CACHE` | Boolean | `0` | Disable the in-process compiled-model / decoder cache (cache is on by default). Set to `1` to disable. |723| `GGML_OPENVINO_DISABLE_KV_SLICE` | Boolean | `0` | Disable the KV-cache input-tensor slicing optimization (slicing is on by default on CPU/GPU). Set to `1` to disable. |724| `GGML_OPENVINO_DISABLE_KV_STATE_RELAYOUT` | Boolean | `0` | Disable the stateful KV-state sequence-axis relayout (relayout is on by default). It moves the KV state sequence axis from dim 1 to dim 2, so the GPU plugin can append new tokens in place instead of copying the whole state every token, and the reader side no longer transposes the whole accumulated state. Set to `1` to disable. |725| `GGML_OPENVINO_MANUAL_GQA_ATTN` | Boolean | device-based | Tri-state. When **unset**, manual GQA attention is enabled by default on `GPU` and disabled on other devices. Set to a positive integer to force-enable, or `0` to force-disable. |726| `GGML_OPENVINO_MEMORY_OPTIMIZE` | Boolean | `0` | Umbrella switch for compile-time memory reductions. Enables `GGML_OPENVINO_REDUCE_COMPILE_MEM` and, on GPU, `GGML_OPENVINO_RELEASE_WEIGHTS` unless those fine-grained variables are explicitly set. |727| `GGML_OPENVINO_REDUCE_COMPILE_MEM`| Boolean | inherits from `GGML_OPENVINO_MEMORY_OPTIMIZE` | Reduce compile-time host memory use by streaming weight requantization and avoiding extra weight-node materialization where possible. Set explicitly to override the umbrella switch. |728| `GGML_OPENVINO_RELEASE_WEIGHTS` | Boolean | inherits from `GGML_OPENVINO_MEMORY_OPTIMIZE` on GPU | GPU-only. Release host weight buffers after the compiled model cache can reuse the device/plugin copy. Requires stable graph shapes; dynamic workloads that need recompilation should leave this disabled. |729| `GGML_OPENVINO_SPILL_DIR` | String | `not set` | Directory for a disk-backed weight buffer. When set, the repacked weight buffer is mapped from an unlinked file on this path instead of anonymous memory, so its pages are reclaimable under memory pressure instead of staying pinned, cutting the load-time host memory peak. Must point at real storage; a tmpfs mount (e.g. `/tmp` on many systems) backs it with RAM and makes the peak worse. |730| `GGML_OPENVINO_REQUANT_KQUANT` | String | `not set` | Requantize Q6_K/Q5_K weights (and matching MoE expert weights) to a 4-bit target instead of the default Q8_0_C, trading accuracy for less memory traffic. One of `q4_sym128` (Q6_K/Q5_K only), `q4_sym128_all` (Q4_K too, drops its per-group zero point), `q4_asym64_all` (Q6_K/Q5_K/Q4_K, keeps a real zero point at group 64), or `native` (no requantization). |731| `GGML_OPENVINO_PROFILING` | Boolean | `0` | Enable execution-time profiling. |732| `GGML_OPENVINO_DUMP_CGRAPH` | Boolean | `0` | Dump the GGML compute graph to `cgraph_ov.txt`. |733| `GGML_OPENVINO_DUMP_IR` | Boolean | `0` | Serialize OpenVINO IR files with timestamps. |734| `GGML_OPENVINO_DEBUG_INPUT` | Boolean | `0` | Enable input debugging and print input tensor info. |735| `GGML_OPENVINO_DEBUG_OUTPUT` | Boolean | `0` | Enable output debugging and print output tensor info. |736| `GGML_OPENVINO_PRINT_CGRAPH_TENSOR_ADDRESS` | Boolean | `0` | Print tensor address map once. |737| `GGML_OPENVINO_LOG_UNSUPPORTED_OPS`| Boolean | `0` | Log warning messages with tensor details and rejection reasons for any ops not supported by the OpenVINO backend. Emits at `WARN` level (requires `--log-verbosity >= 2`, enabled by default). |738 739> [!NOTE]740> - `GGML_OPENVINO_STATEFUL_EXECUTION` is an **Experimental** feature to allow stateful execution for managing the KV cache internally inside the OpenVINO model, improving performance on CPUs and GPUs. Stateful execution is not effective on NPUs, and not all models currently support this feature. This feature is experimental and has been validated only with the llama-simple, llama-cli, llama-bench, and llama-run applications and is recommended to enable for the best performance. Other applications, such as llama-server and llama-perplexity, are not yet supported.741> - `GGML_OPENVINO_LOG_UNSUPPORTED_OPS` emits logs at `WARN` level (`GGML_LOG_WARN`), which requires application log verbosity `--log-verbosity >= 2` (or `-lv 2`).742 743### Example Usage744 745#### GPU Inference with Profiling746 747```bash748# If the system has multiple GPUs, use GPU.0 or GPU.1 to explicitly target a specific GPU.749 750# Linux751export GGML_OPENVINO_CACHE_DIR=/tmp/ov_cache752export GGML_OPENVINO_PROFILING=1753export GGML_OPENVINO_DEVICE=GPU754export GGML_OPENVINO_STATEFUL_EXECUTION=1755 756./build/ReleaseOV/bin/llama-simple -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -n 50 "The story of AI is "757 758# Windows Command Line759set GGML_OPENVINO_CACHE_DIR=C:\tmp\ov_cache760set GGML_OPENVINO_PROFILING=1761set GGML_OPENVINO_DEVICE=GPU762set GGML_OPENVINO_STATEFUL_EXECUTION=1763 764# Windows PowerShell765$env:GGML_OPENVINO_CACHE_DIR = "C:\tmp\ov_cache"766$env:GGML_OPENVINO_PROFILING = "1"767$env:GGML_OPENVINO_DEVICE = "GPU"768$env:GGML_OPENVINO_STATEFUL_EXECUTION = "1"769 770build\ReleaseOV\bin\llama-simple.exe -m "C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf" -n 50 "The story of AI is "771 772```773 774## Known Limitations775 776**General (all devices)**777 778- Llama.cpp OpenVINO backend currently supports a subset of GGML ops and text-only models. Unsupported ops or unsupported op shapes/cases fail during OpenVINO translation.779- Multimodal features (audio/image/video) are a work in progress.780- Limited Embedding and Reranking model support.781- Llama.cpp tool coverage across CPU/GPU/NPU is not uniform.782 783**Tool-specific**784 785- `llama-bench`: requires `-fa 1` (flash-attention).786- `llama-cli --context-shift`: stateless only (`GGML_OPENVINO_STATEFUL_EXECUTION=0`). In stateful mode the KV cache is owned by the OpenVINO model and cannot be shifted externally.787- `llama-server`: only one chat session/thread when `GGML_OPENVINO_STATEFUL_EXECUTION=1`.788 789**GPU-specific**790 791- `llama-server -np > 1`: concurrent requests are batched together, which may slightly reduce per-request throughput.792 793**NPU-specific**794 795- Default context resolves to the model's training context (e.g. 131072 for Llama 3.2 1B), which can OOM or fail or degrade performance on NPU. Inspect the resolved value with `-lv 3`.796 - **Workaround:** Pass an explicit `-c <N>`, e.g. `-c 1024`.797- NPU device uses a static graph with a fixed prefill chunk size (defaults to 256), configurable with `GGML_OPENVINO_PREFILL_CHUNK_SIZE`. Large prefill/batch settings may need tuning.798- `llama-server -np > 1` (multiple parallel sequences) is not supported.799- `llama-perplexity`: requires `-b 512` or smaller.800 801> [!NOTE]802> The OpenVINO backend is actively under development. Fixes and improvements are underway, and this document will continue to be updated.803 804## Work in Progress805 806- Performance and memory optimizations807- Accuracy validation808- Broader quantization coverage809- Support for additional model architectures810