Felipe97/llama-cpp-compiled

1# Multimodal2 3llama.cpp supports multimodal input via `libmtmd`. Currently, there are 2 tools support this feature:4- [llama-cli](../tools/cli/README.md)5- [llama-server](../tools/server/README.md) via OpenAI-compatible `/chat/completions` API6- [llama-mtmd-cli](../tools/mtmd/README.md), for testing and development7 8Currently, we support **image**, **audio** and **video** input.9 10To enable it, you can use one of the 2 methods below:11 12- Use `-hf` option with a supported model (see a list of pre-quantized model below)13    - To load a model using `-hf` while disabling multimodal, use `--no-mmproj`14    - To load a model using `-hf` while using a custom mmproj file, use `--mmproj local_file.gguf`15- Use `-m model.gguf` option with `--mmproj file.gguf` to specify text and multimodal projector respectively16 17By default, multimodal projector will be offloaded to GPU. To disable this, add `--no-mmproj-offload`18 19For example:20 21```sh22# simple usage with CLI23llama-mtmd-cli -hf ggml-org/gemma-3-4b-it-GGUF24 25# simple usage with server26llama-server -hf ggml-org/gemma-3-4b-it-GGUF27 28# using local file29llama-server -m gemma-3-4b-it-Q4_K_M.gguf --mmproj mmproj-gemma-3-4b-it-Q4_K_M.gguf30 31# no GPU offload32llama-server -hf ggml-org/gemma-3-4b-it-GGUF --no-mmproj-offload33```34 35> [!IMPORTANT]36>37> OCR models are trained with specific prompt and input structure, please refer to these discussions for more info:38> - PaddleOCR-VL: https://github.com/ggml-org/llama.cpp/pull/1882539> - GLM-OCR: https://github.com/ggml-org/llama.cpp/pull/1967740> - Deepseek-OCR: https://github.com/ggml-org/llama.cpp/pull/1740041> - Dots.OCR: https://github.com/ggml-org/llama.cpp/pull/1757542> - HunyuanOCR: https://github.com/ggml-org/llama.cpp/pull/2139543 44## Pre-quantized models45 46These are ready-to-use models, most of them come with `Q4_K_M` quantization by default. They can be found at the Hugging Face page of the ggml-org: https://huggingface.co/collections/ggml-org/multimodal-ggufs-68244e01ff1f39e5bebeeedc47 48Replaces the `(tool_name)` with the name of binary you want to use. For example, `llama-mtmd-cli` or `llama-server`49 50NOTE: some models may require large context window, for example: `-c 8192`51 52**Vision models**:53 54```sh55# Gemma 356(tool_name) -hf ggml-org/gemma-3-4b-it-GGUF57(tool_name) -hf ggml-org/gemma-3-12b-it-GGUF58(tool_name) -hf ggml-org/gemma-3-27b-it-GGUF59 60# SmolVLM61(tool_name) -hf ggml-org/SmolVLM-Instruct-GGUF62(tool_name) -hf ggml-org/SmolVLM-256M-Instruct-GGUF63(tool_name) -hf ggml-org/SmolVLM-500M-Instruct-GGUF64(tool_name) -hf ggml-org/SmolVLM2-2.2B-Instruct-GGUF65(tool_name) -hf ggml-org/SmolVLM2-256M-Video-Instruct-GGUF66(tool_name) -hf ggml-org/SmolVLM2-500M-Video-Instruct-GGUF67 68# Pixtral 12B69(tool_name) -hf ggml-org/pixtral-12b-GGUF70 71# Qwen 2 VL72(tool_name) -hf ggml-org/Qwen2-VL-2B-Instruct-GGUF73(tool_name) -hf ggml-org/Qwen2-VL-7B-Instruct-GGUF74 75# Qwen 2.5 VL76(tool_name) -hf ggml-org/Qwen2.5-VL-3B-Instruct-GGUF77(tool_name) -hf ggml-org/Qwen2.5-VL-7B-Instruct-GGUF78(tool_name) -hf ggml-org/Qwen2.5-VL-32B-Instruct-GGUF79(tool_name) -hf ggml-org/Qwen2.5-VL-72B-Instruct-GGUF80 81# Mistral Small 3.1 24B (IQ2_M quantization)82(tool_name) -hf ggml-org/Mistral-Small-3.1-24B-Instruct-2503-GGUF83 84# InternVL 2.5 and 385(tool_name) -hf ggml-org/InternVL2_5-1B-GGUF86(tool_name) -hf ggml-org/InternVL2_5-4B-GGUF87(tool_name) -hf ggml-org/InternVL3-1B-Instruct-GGUF88(tool_name) -hf ggml-org/InternVL3-2B-Instruct-GGUF89(tool_name) -hf ggml-org/InternVL3-8B-Instruct-GGUF90(tool_name) -hf ggml-org/InternVL3-14B-Instruct-GGUF91 92# Llama 4 Scout93(tool_name) -hf ggml-org/Llama-4-Scout-17B-16E-Instruct-GGUF94 95# Moondream2 20250414 version96(tool_name) -hf ggml-org/moondream2-20250414-GGUF97 98# Gemma 499(tool_name) -hf ggml-org/gemma-4-E2B-it-GGUF100(tool_name) -hf ggml-org/gemma-4-E4B-it-GGUF101(tool_name) -hf ggml-org/gemma-4-26B-A4B-it-GGUF102(tool_name) -hf ggml-org/gemma-4-31B-it-GGUF103```104 105**Audio models**:106 107```sh108# Ultravox 0.5109(tool_name) -hf ggml-org/ultravox-v0_5-llama-3_2-1b-GGUF110(tool_name) -hf ggml-org/ultravox-v0_5-llama-3_1-8b-GGUF111 112# Qwen2-Audio and SeaLLM-Audio113# note: no pre-quantized GGUF this model, as they have very poor result114# ref: https://github.com/ggml-org/llama.cpp/pull/13760115 116# Mistral's Voxtral117(tool_name) -hf ggml-org/Voxtral-Mini-3B-2507-GGUF118 119# Qwen3-ASR120(tool_name) -hf ggml-org/Qwen3-ASR-0.6B-GGUF121(tool_name) -hf ggml-org/Qwen3-ASR-1.7B-GGUF122```123 124**Mixed modalities**:125 126```sh127# Qwen2.5 Omni128# Capabilities: audio input, vision input129(tool_name) -hf ggml-org/Qwen2.5-Omni-3B-GGUF130(tool_name) -hf ggml-org/Qwen2.5-Omni-7B-GGUF131 132# Qwen3 Omni133# Capabilities: audio input, vision input134(tool_name) -hf ggml-org/Qwen3-Omni-30B-A3B-Instruct-GGUF135(tool_name) -hf ggml-org/Qwen3-Omni-30B-A3B-Thinking-GGUF136 137# Gemma 4138# Capabilities: audio input, vision input139(tool_name) -hf ggml-org/gemma-4-E2B-it-GGUF140(tool_name) -hf ggml-org/gemma-4-E4B-it-GGUF141```142 143## Finding more models:144 145GGUF models on Huggingface with vision capabilities can be found here: https://huggingface.co/models?pipeline_tag=image-text-to-text&sort=trending&search=gguf146