Felipe97/llama-cpp-compiled

1# Model Conversion Example2This directory contains scripts and code to help in the process of converting3HuggingFace PyTorch models to GGUF format.4 5The motivation for having this is that the conversion process can often be an6iterative process, where the original model is inspected, converted, updates7made to llama.cpp, converted again, etc. Once the model has been converted it8needs to be verified against the original model, and then optionally quantified,9and in some cases perplexity checked of the quantized model. And finally the10model/models need to the ggml-org on Hugging Face. This tool/example tries to11help with this process.12 13> 📝 **Note:** When adding a new model from an existing family, verify the14> previous version passes logits verification first. Existing models can have15> subtle numerical differences that don't affect generation quality but cause16> logits mismatches. Identifying these upfront whether they exist in llama.cpp,17> the conversion script, or in an upstream implementation, can save significant18> debugging time.19 20### Overview21The idea is that the makefile targets and scripts here can be used in the22development/conversion process assisting with things like:23 24* inspect/run the original model to figure out how it works25* convert the original model to GGUF format26* inspect/run the converted model27* verify the logits produced by the original model and the converted model28* quantize the model to GGUF format29* run perplexity evaluation to verify that the quantized model is performing30  as expected31* upload the model to HuggingFace to make it available for others32 33## Setup34Create virtual python environment35```console36$ python3.11 -m venv venv37$ source venv/bin/activate38(venv) $ pip install -r requirements.txt39```40 41## Causal Language Model Conversion42This section describes the steps to convert a causal language model to GGUF and43to verify that the conversion was successful.44 45### Download the original model46First, clone the original model to some local directory:47```console48$ mkdir models && cd models49$ git clone https://huggingface.co/user/model_name50$ cd model_name51$ git lfs install52$ git lfs pull53```54 55### Set the MODEL_PATH56The path to the downloaded model can be provided in two ways:57 58**Option 1: Environment variable (recommended for iterative development)**59```console60export MODEL_PATH=~/work/ai/models/some_model61```62 63**Option 2: Command line argument (for one-off tasks)**64```console65make causal-convert-model MODEL_PATH=~/work/ai/models/some_model66```67 68Command line arguments take precedence over environment variables when both are provided.69 70In cases where the transformer implementation for the model has not been released71yet it is possible to set the environment variable `UNRELEASED_MODEL_NAME` which72will then cause the transformer implementation to be loaded explicitly and not73use AutoModelForCausalLM:74```75export UNRELEASED_MODEL_NAME=SomeNewModel76```77 78### Inspecting the original tensors79```console80# Using environment variable81(venv) $ make causal-inspect-original-model82 83# Or using command line argument84(venv) $ make causal-inspect-original-model MODEL_PATH=~/work/ai/models/some_model85```86 87### Running the original model88This is mainly to verify that the original model works, and to compare the output89from the converted model.90```console91# Using environment variable92(venv) $ make causal-run-original-model93 94# Or using command line argument95(venv) $ make causal-run-original-model MODEL_PATH=~/work/ai/models/some_model96```97This command will save two files to the `data` directory, one is a binary file98containing logits which will be used for comparison with the converted model99later, and the other is a text file which allows for manual visual inspection.100 101### Model conversion102After updates have been made to [gguf-py](../../gguf-py) to add support for the103new model, the model can be converted to GGUF format using the following command:104```console105# Using environment variable106(venv) $ make causal-convert-model107 108# Or using command line argument109(venv) $ make causal-convert-model MODEL_PATH=~/work/ai/models/some_model110```111 112### Inspecting the converted model113The converted model can be inspected using the following command:114```console115(venv) $ make causal-inspect-converted-model116```117 118### Running the converted model119```console120(venv) $ make causal-run-converted-model121```122 123### Model logits verification124The following target will run the original model and the converted model and125compare the logits:126```console127(venv) $ make causal-verify-logits128```129 130### Quantizing the model131The causal model can be quantized to GGUF format using the following command:132```console133(venv) $ make causal-quantize-Q8_0134Quantized model saved to: /path/to/quantized/model-Q8_0.gguf135Export the quantized model path to QUANTIZED_MODEL variable in your environment136```137This will show the path to the quantized model in the terminal, which can then138be used to set the `QUANTIZED_MODEL` environment variable:139```console140export QUANTIZED_MODEL=/path/to/quantized/model-Q8_0.gguf141```142Then the quantized model can be run using the following command:143```console144(venv) $ make causal-run-quantized-model145```146 147### Quantizing QAT (Quantization Aware Training) models148When quantizing to `Q4_0`, the default data type for the token embedding weights149will be `Q6_K`. For models that are going to be uploaded to ggml-org it is150recommended to use `Q8_0` instead for the embeddings and output tensors.151The reason is that although `Q6_K` is smaller in size, it requires more compute152to unpack, which can hurt performance during output generation when the entire153embedding matrix must be dequantized to compute vocabulary logits. `Q8_0`154provides practically full quality with better computational efficiency.155```console156(venv) $ make causal-quantize-qat-Q4_0157```158 159 160## Embedding Language Model Conversion161 162### Download the original model163```console164$ mkdir models && cd models165$ git clone https://huggingface.co/user/model_name166$ cd model_name167$ git lfs install168$ git lfs pull169```170 171The path to the embedding model can be provided in two ways:172 173**Option 1: Environment variable (recommended for iterative development)**174```console175export EMBEDDING_MODEL_PATH=~/path/to/embedding_model176```177 178**Option 2: Command line argument (for one-off tasks)**179```console180make embedding-convert-model EMBEDDING_MODEL_PATH=~/path/to/embedding_model181```182 183Command line arguments take precedence over environment variables when both are provided.184 185### Running the original model186This is mainly to verify that the original model works and to compare the output187with the output from the converted model.188```console189# Using environment variable190(venv) $ make embedding-run-original-model191 192# Or using command line argument193(venv) $ make embedding-run-original-model EMBEDDING_MODEL_PATH=~/path/to/embedding_model194```195This command will save two files to the `data` directory, one is a binary196file containing logits which will be used for comparison with the converted197model, and the other is a text file which allows for manual visual inspection.198 199#### Using SentenceTransformer with numbered layers200For models that have numbered SentenceTransformer layers (01_Pooling, 02_Dense,20103_Dense, 04_Normalize), these will be applied automatically when running the202converted model but currently there is a separate target to run the original203version:204 205```console206# Run original model with SentenceTransformer (applies all numbered layers)207(venv) $ make embedding-run-original-model-st208```209 210This will use the SentenceTransformer library to load and run the model, which211automatically applies all the numbered layers in the correct order. This is212particularly useful when comparing with models that should include these213additional transformation layers beyond just the base model output.214 215The type of normalization can be specified for the converted model but is not216strictly necessary as the verification uses cosine similarity and the magnitude217of the output vectors does not affect this. But the normalization type can be218specified as an argument to the target which might be useful for manual219inspection:220```console221(venv) $ make embedding-verify-logits-st EMBD_NORMALIZE=1222```223The original model will apply the normalization according to the normalization224layer specified in the modules.json configuration file.225 226### Model conversion227After updates have been made to [gguf-py](../../gguf-py) to add support for the228new model the model can be converted to GGUF format using the following command:229```console230(venv) $ make embedding-convert-model231```232 233### Run the converted model234```console235(venv) $ make embedding-run-converted-model236```237 238### Model logits verification239The following target will run the original model and the converted model (which240was done manually in the previous steps) and compare the logits:241```console242(venv) $ make embedding-verify-logits243```244 245For models with SentenceTransformer layers, use the `-st` verification target:246```console247(venv) $ make embedding-verify-logits-st248```249This convenience target automatically runs both the original model with SentenceTransformer250and the converted model with pooling enabled, then compares the results.251 252### llama-server verification253To verify that the converted model works with llama-server, the following254command can be used:255```console256(venv) $ make embedding-start-embedding-server257```258Then open another terminal and set the `EMBEDDINGS_MODEL_PATH` environment259variable as this will not be inherited by the new terminal:260```console261(venv) $ make embedding-curl-embedding-endpoint262```263This will call the `embedding` endpoing and the output will be piped into264the same verification script as used by the target `embedding-verify-logits`.265 266The causal model can also be used to produce embeddings and this can be verified267using the following commands:268```console269(venv) $ make causal-start-embedding-server270```271Then open another terminal and set the `MODEL_PATH` environment272variable as this will not be inherited by the new terminal:273```console274(venv) $ make casual-curl-embedding-endpoint275```276 277### Quantizing the model278The embedding model can be quantized to GGUF format using the following command:279```console280(venv) $ make embedding-quantize-Q8_0281Quantized model saved to: /path/to/quantized/model-Q8_0.gguf282Export the quantized model path to QUANTIZED_EMBEDDING_MODEL variable in your environment283```284This will show the path to the quantized model in the terminal, which can then285be used to set the `QUANTIZED_EMBEDDING_MODEL` environment variable:286```console287export QUANTIZED_EMBEDDING_MODEL=/path/to/quantized/model-Q8_0.gguf288```289Then the quantized model can be run using the following command:290```console291(venv) $ make embedding-run-quantized-model292```293 294### Quantizing QAT (Quantization Aware Training) models295When quantizing to `Q4_0`, the default data type for the token embedding weights296will be `Q6_K`. For models that are going to be uploaded to ggml-org it is297recommended to use `Q8_0` instead for the embeddings and output tensors.298The reason is that although `Q6_K` is smaller in size, it requires more compute299to unpack, which can hurt performance during output generation when the entire300embedding matrix must be dequantized to compute vocabulary logits. `Q8_0`301provides practically full quality with better computational efficiency.302```console303(venv) $ make embedding-quantize-qat-Q4_0304```305 306## Perplexity Evaluation307 308### Simple perplexity evaluation309This allows to run the perplexity evaluation without having to generate a310token/logits file:311```console312(venv) $ make perplexity-run QUANTIZED_MODEL=~/path/to/quantized/model.gguf313```314This will use the wikitext dataset to run the perplexity evaluation and315output the perplexity score to the terminal. This value can then be compared316with the perplexity score of the unquantized model.317 318### Full perplexity evaluation319First use the converted, non-quantized, model to generate the perplexity evaluation320dataset using the following command:321```console322$ make perplexity-data-gen CONVERTED_MODEL=~/path/to/converted/model.gguf323```324This will generate a file in the `data` directory named after the model and with325a `.kld` suffix which contains the tokens and the logits for the wikitext dataset.326 327After the dataset has been generated, the perplexity evaluation can be run using328the quantized model:329```console330$ make perplexity-run-full QUANTIZED_MODEL=~/path/to/quantized/model-Qxx.gguf LOGITS_FILE=data/model.gguf.ppl331```332 333> 📝 **Note:** The `LOGITS_FILE` is the file generated by the previous command334> can be very large, so make sure you have enough disk space available.335 336## HuggingFace utilities337The following targets are useful for creating collections and model repositories338on Hugging Face in the ggml-org. These can be used when preparing a release339to script the process for new model releases.340 341For the following targets a `HF_TOKEN` environment variable is required.342 343> 📝 **Note:** Don't forget to logout from Hugging Face after running these344> commands, otherwise you might have issues pulling/cloning repositories as345> the token will still be in use:346> $ huggingface-cli logout347> $ unset HF_TOKEN348 349### Create a new Hugging Face Model (model repository)350This will create a new model repository on Hugging Face with the specified351model name.352```console353(venv) $ make hf-create-model MODEL_NAME='TestModel' NAMESPACE="danbev" ORIGINAL_BASE_MODEL="some-base-model"354Repository ID:  danbev/TestModel-GGUF355Repository created: https://huggingface.co/danbev/TestModel-GGUF356```357Note that we append a `-GGUF` suffix to the model name to ensure a consistent358naming convention for GGUF models.359 360An embedding model can be created using the following command:361```console362(venv) $ make hf-create-model-embedding MODEL_NAME='TestEmbeddingModel' NAMESPACE="danbev" ORIGINAL_BASE_MODEL="some-base-model"363```364The only difference is that the model card for an embedding model will be different365with regards to the llama-server command and also how to access/call the embedding366endpoint.367 368### Upload a GGUF model to model repository369The following target uploads a model to an existing Hugging Face model repository.370```console371(venv) $ make hf-upload-gguf-to-model MODEL_PATH=dummy-model1.gguf REPO_ID=danbev/TestModel-GGUF372📤 Uploading dummy-model1.gguf to danbev/TestModel-GGUF/dummy-model1.gguf373✅ Upload successful!374🔗 File available at: https://huggingface.co/danbev/TestModel-GGUF/blob/main/dummy-model1.gguf375```376This command can also be used to update an existing model file in a repository.377 378### Create a new Collection379```console380(venv) $ make hf-new-collection NAME=TestCollection DESCRIPTION="Collection for testing scripts" NAMESPACE=danbev381🚀 Creating Hugging Face Collection382Title: TestCollection383Description: Collection for testing scripts384Namespace: danbev385Private: False386✅ Authenticated as: danbev387📚 Creating collection: 'TestCollection'...388✅ Collection created successfully!389📋 Collection slug: danbev/testcollection-68930fcf73eb3fc200b9956d390🔗 Collection URL: https://huggingface.co/collections/danbev/testcollection-68930fcf73eb3fc200b9956d391 392🎉 Collection created successfully!393Use this slug to add models: danbev/testcollection-68930fcf73eb3fc200b9956d394```395 396### Add model to a Collection397```console398(venv) $ make hf-add-model-to-collection COLLECTION=danbev/testcollection-68930fcf73eb3fc200b9956d MODEL=danbev/TestModel-GGUF399✅ Authenticated as: danbev400🔍 Checking if model exists: danbev/TestModel-GGUF401✅ Model found: danbev/TestModel-GGUF402📚 Adding model to collection...403✅ Model added to collection successfully!404🔗 Collection URL: https://huggingface.co/collections/danbev/testcollection-68930fcf73eb3fc200b9956d405 406🎉 Model added successfully!407 408```409