Felipe97/llama-cpp-compiled

1# Add a new model architecture to `llama.cpp`2 3Adding a model requires few steps:4 51. Convert the model to GGUF62. Define the model architecture in `llama.cpp`73. Build the GGML graph implementation84. Optional: Add multimodal encoder implementation9 10After following these steps, you can open PR.11 12Also, it is important to check that the examples and main ggml backends (CUDA, METAL, CPU) are working with the new architecture, especially:13- [cli](/tools/cli/)14- [completion](/tools/completion/)15- [imatrix](/tools/imatrix/)16- [quantize](/tools/quantize/)17- [server](/tools/server/)18 19### 1. Convert the model to GGUF20 21This step is done in python with a `convert` script using the [gguf](https://pypi.org/project/gguf/) library.22Depending on the model architecture, you can use either [convert_hf_to_gguf.py](/convert_hf_to_gguf.py) or [examples/convert_legacy_llama.py](/examples/convert_legacy_llama.py) (for `llama/llama2` models in `.pth` format).23 24The convert script reads the model configuration, tokenizer, tensor names+data and converts them to GGUF metadata and tensors.25 26The required steps to implement for an HF model are:27 281. Define the model `ModelBase.register` annotation in a new `TextModel` or `MmprojModel` subclass in the [conversion](/conversion) folder, example:29 30```python31@ModelBase.register("MyModelForCausalLM")32@ModelBase.example("user/model")33class MyModel(TextModel):34    model_arch = gguf.MODEL_ARCH.MYMODEL35```36 37or38 39```python40@ModelBase.register("MyModelForConditionalGeneration")41@ModelBase.example("user/model")42class MyModel(MmprojModel):43    model_arch = gguf.MODEL_ARCH.MYMODEL44```45 46The `example` should point to a valid Hugging Face model that will be used for testing. You can add multiple models if necessary. Prefer a non-gated model, or tiny random weights if no such model exists.47 482. Define the layout of the GGUF tensors in [constants.py](/gguf-py/gguf/constants.py)49 50Add an enum entry in `MODEL_ARCH`, the model human friendly name in `MODEL_ARCH_NAMES` and the GGUF tensor names in `MODEL_TENSORS`.51 52NOTE: Pick the GGUF arch string (and the matching `src/models/<name>.cpp` filename, see section 3) carefully up front, following existing naming conventions. Once GGUF files are published under a given arch string, renaming it later breaks the community's existing files, so this is not something to leave for cleanup in a follow-up PR.53 54Example for `falcon` model:55```python56    MODEL_ARCH.FALCON: [57        MODEL_TENSOR.TOKEN_EMBD,58        MODEL_TENSOR.OUTPUT_NORM,59        MODEL_TENSOR.OUTPUT,60        MODEL_TENSOR.ATTN_NORM,61        MODEL_TENSOR.ATTN_NORM_2,62        MODEL_TENSOR.ATTN_QKV,63        MODEL_TENSOR.ATTN_OUT,64        MODEL_TENSOR.FFN_DOWN,65        MODEL_TENSOR.FFN_UP,66    ]67```68 693. Map the original tensor names to the standardize equivalent in GGUF70 71As a general rule, before adding a new tensor name to GGUF, be sure the equivalent naming does not already exist.72 73Once you have found the GGUF tensor name equivalent, add it to the [tensor_mapping.py](/gguf-py/gguf/tensor_mapping.py) file.74 75If the tensor name is part of a repetitive layer/block, the key word `bid` substitutes it.76 77Example for the normalization tensor in attention layers:78 79```python80block_mappings_cfg: dict[MODEL_TENSOR, tuple[str, ...]] = {81        # Attention norm82        MODEL_TENSOR.ATTN_NORM: (83            "gpt_neox.layers.{bid}.input_layernorm",                # gptneox84            "transformer.h.{bid}.ln_1",                             # gpt2 gpt-j refact qwen85            "transformer.blocks.{bid}.norm_1",                      # mpt86            ...87        )88}89```90 91`transformer.blocks.{bid}.norm_1` will be mapped to `blk.{bid}.attn_norm` in GGUF.92 93Depending on the model configuration, tokenizer, code and tensors layout, you will have to override:94- `TextModel#set_gguf_parameters`95- `MmprojModel#set_gguf_parameters`96- `ModelBase#set_vocab`97- `ModelBase#modify_tensors`98 99NOTE: Tensor names must end with `.weight` or `.bias` suffixes, that is the convention and several tools like `quantize` expect this to proceed the weights.100 101### 2. Define the model architecture in `llama.cpp`102 103The model params and tensors layout must be defined in `llama.cpp` source files:1041. Define a new `llm_arch` enum value in `src/llama-arch.h`.1052. In `src/llama-arch.cpp`:106    - Add the architecture name to the `LLM_ARCH_NAMES` map.107    - You may also need to update `LLM_KV_NAMES`, `LLM_TENSOR_NAMES` and `LLM_TENSOR_INFOS`1083. Add any non-standard metadata loading in the `llama_model_loader` constructor in `src/llama-model-loader.cpp`.1094. If the model has a RoPE operation, add a case for the architecture in `llama_model_rope_type` function in `src/llama-model.cpp`.1105. Check for other places that switch/iterate over every `llm_arch` value, e.g. `src/llama-model-saver.cpp` and any mandatory-hparam lists (such as which archs require MoE metadata). Grep for `LLM_ARCH_` usages to find them. Missing one of these is a common cause of CI test failures (e.g. `test-llama-archs`) after adding a new arch.111 112NOTE: The dimensions in `ggml` are typically in the reverse order of the `pytorch` dimensions.113 114### 3. Build the GGML graph implementation115 116This is the funniest part, you have to provide the inference graph implementation of the new model architecture in `src/llama-model.cpp`:1171. Create a new struct that inherits from `llama_model_base`.1182. Implement the graph-building logic in its `build_arch_graph` method.1193. The `build_arch_graph` method should return a constructed graph (inherited from `llm_graph_context`). Have a look at existing implementations like `llama_model_llama`, `llama_model_dbrx` or `llama_model_bert`.1204. Then, in the `llama_model_mapping` function, add a case for your architecture to instantiate your new graph-building struct.121 122Some `ggml` backends do not support all operations. Backend implementations can be added in a separate PR.123 124Note: to debug the inference graph: you can use [llama-eval-callback](/examples/eval-callback/).125 126### 4. Optional: Add multimodal encoder implementation127 128If the new model supports multimodal inputs, you will need to add a new encoder definition in `libmtmd`. You can find more information about llama.cpp's multimodal support in [the docs](../multimodal.md) and in the `tools/mtmd` source directory.129 1301. In the conversion script, make sure you add a subclass that extends `MmprojModel` or another class that inherits from the same base class.1312. Add the encoder definition in `clip.cpp`.1323. Implement the preprocessor in `mtmd.cpp`. In most cases, you can reuse an existing preprocessor.1334. Implement the encoder GGML graph, either in a dedicated file if the model is truly different from existing ones, or by reusing an existing implementation (for example: siglip, pixtral, or qwen) and adding a model-specific projector.134 135Note:136- Many multimodal encoders are based on models that are already supported. Make sure to read the existing encoder definitions in `tools/mtmd/models` before adding a new one. In `libmtmd`, it is generally better to extend an existing model than to duplicate code.137- To debug the multimodal preprocessor and encoder, you can use [llama-mtmd-debug](tools/mtmd/debug/mtmd-debug.cpp).138- Adding a model-specific API or CLI is an anti-pattern in `libmtmd`. The goal of `libmtmd` is to provide an easy-to-use, model-agnostic library for multimodal pipeline.139- In most cases, `llama-mtmd-cli` should not be modified. If a model requires a specific prompt, either let the user provide it or bake it into the Jinja chat template.140- For audio generation models, see `tools/mtmd/README-dev.md`141 142## Tips and tricks143 144### Prefer conversion-time tensor modifications over graph-time ones145 146If the model contains constant modifications of tensors in the graph (for example, `norm(1 + weight)`) or performs tensor permutations/chunking, perform the modifications during conversion rather than in the graph code. This keeps the inference graph simpler and avoids extra runtime ops.147 148Examples:149- Gemma 3 folds the `1 +` of its `norm(1 + weight)` normalization into the weights at conversion time, so the graph just does a plain RMS norm.150- Qwen3-Next applies its tensor permutation during conversion (in `modify_tensors`), so the graph can consume the already-permuted weights directly.151 152Exception: a plain `weight * scale` with a constant scale is usually better left to inference time rather than folded into the weight at conversion. The scale conceptually applies to the activation, not the weight, so folding it into the weight can hurt numerical stability, and it shifts the weight's value range in a way that can make quantization worse. In this case, write the scale to GGUF as its own metadata key (e.g. `%s.attention.output_scale`, `%s.attention.value_scale`, `%s.embedding_scale`) and apply it in the graph, instead of pre-multiplying the weight tensor during conversion.153 154### Working with ggml_rope_ext155 156PyTorch implementations usually prefer explicitly calculating `freq_cis`/`sin`/`cos` components. However, in llama.cpp, most RoPE operations can be handled via `ggml_rope_ext`, which does not require a sin/cos matrix. This saves memory while allowing the GGML RoPE kernel to be fused with other ops.157 158However, since `ggml_rope_ext` only provides a subset of the RoPE implementations that models use, converting models from PyTorch to llama.cpp may require some creative adaptations.159 160For more information about `ggml_rope_ext`, please refer to the in-code documentation in `ggml.h`.161 162Examples:163- `libmtmd` implements 2D RoPE with `GGML_ROPE_TYPE_NORMAL` ordering by splitting the input tensor in half, applying `ggml_rope_ext` separately to each half, then joining them back together using `ggml_concat`.164- The [Kimi-K2.5](https://github.com/ggml-org/llama.cpp/pull/19170) vision encoder uses vision RoPE with interleaved frequencies. The weights must be permuted during conversion in order to reuse the `build_rope_2d()` function.165- [Gemma 4](https://github.com/ggml-org/llama.cpp/pull/21309) uses "proportional" RoPE. We employ a trick where `rope_freqs` is set to a very large value in the last dimensions to prevent those dimensions from being rotated. See the `Gemma4Model` class in `convert_hf_to_gguf.py`.166- Some models require scaling the input position. For example, `[0, 1, 2, ...]` becomes `[0, 0.5, 1, ...]`. In this case, you can provide the scaling via `freq_scale = 0.5f`.167- Some models use learned RoPE frequencies instead of relying on `powf(freq_base, -2.0 * i / n_dims)`. In this case, you can provide the learned frequencies via the `rope_freqs` tensor (corresponding to the `c` argument in `ggml_rope_ext`), then set `freq_base = 1.0f`. An important note is that `rope_freqs` in GGML is the **inverse** (`theta = pos[i] / rope_freqs`), so you may need to invert `rope_freqs` during conversion.168 169### Rotating only a part of the head170 171Many models rotate only a part of each head and leave the rest untouched (often called the "nope" part). Do not build this with views plus `ggml_concat`, it's not efficient. Both layouts can be done with a single RoPE op:172 173- `[rope|nope]`, rotated dims first: pass `n_dims` smaller than the head size to `ggml_rope_ext`. Dims from `n_dims` to the end are copied as-is.174- `[nope|rope]`, rotated dims last: call `ggml_rope_set_offset(cur, n_offs)` on the result of the RoPE, where `n_offs` is the size of the leading untouched part. Dims outside `[n_offs, n_offs + n_dims)` are copied as-is.175 176`n_offs` must be even, `n_offs + n_dims` must fit in the row, and vision RoPE is not supported. Note that the frequencies are computed relative to the rotated window.177 178Example: DeepSeek-V4 uses `[nope|rope]` for its query, key and compressed KV tensors, so `src/models/deepseek4.cpp` ropes the whole tensor and then calls `ggml_rope_set_offset(cur, n_embd_head_nope)`.179 180Exception: some models apply an extra op to the `nope` part, for example `deepseek32.cpp`, and may not use this optimization. While RoPE can be applied selectively to a part of the head, the extra op may not, so these models still need views plus `ggml_concat`.181 182## GGUF specification183 184https://github.com/ggml-org/ggml/blob/master/docs/gguf.md185 186## Resources187 188- YaRN RoPE scaling https://github.com/ggml-org/llama.cpp/pull/2268189- support Baichuan serial models https://github.com/ggml-org/llama.cpp/pull/3009190- support attention bias https://github.com/ggml-org/llama.cpp/pull/4283191- Mixtral support https://github.com/ggml-org/llama.cpp/pull/4406192- BERT embeddings https://github.com/ggml-org/llama.cpp/pull/5423193- Grok-1 support https://github.com/ggml-org/llama.cpp/pull/6204194- Command R Plus support https://github.com/ggml-org/llama.cpp/pull/6491195- support arch DBRX https://github.com/ggml-org/llama.cpp/pull/6515196- How to convert HuggingFace model to GGUF format https://github.com/ggml-org/llama.cpp/discussions/2948197