Car5577/mlx-to-gguf-apple-silicon

mlx-to-gguf-apple-silicon

Convert an MLX 4-bit fine-tuned model (the output of mlx_lm.fuse) into a standard safetensors model that llama.cpp can convert to GGUF — so you can run your MLX-trained model in Ollama on a Mac.

This solves a long-standing gap for the Apple Silicon local-AI community: fine-tuning with MLX is the natural workflow on a Mac, but the resulting quantized model could not be converted to GGUF for Ollama. The existing solutions (e.g. Unsloth) require an NVIDIA GPU and don't help Mac users.

A note on authorship. I'm not a programmer — I can't write code. This solution came out of a collaboration with Claude (Anthropic): I posed the problem, set the constraints (MacBook Pro M-series, 24 GB RAM) and validated the results at each step; Claude wrote and debugged the code. I'm publishing it so it can help other Apple Silicon users in the same situation.

The problem

When you fine-tune a 4-bit MLX model (e.g. mlx-community/Mistral-Small-3.1-24B-Instruct-2503-4bit) and fuse the LoRA adapters with mlx_lm.fuse, the resulting model cannot be converted by llama.cpp/convert_hf_to_gguf.py. It fails with:

ValueError: Can not map tensor 'model.embed_tokens.biases'

There are two distinct reasons:

1. 1.Tensor naming. MLX names tensors with a language_model.model. prefix instead of the model. prefix llama.cpp expects.
2. 2.MLX quantization tensors. Each quantized weight is stored as a uint32 packed tensor plus separate scales and biases tensors. llama.cpp doesn't know how to interpret these.

The solution

mlx_to_gguf_ready.py does three things, one tensor at a time (so it stays light on RAM — it runs comfortably on a 24 GB Mac):

1. 1.Dequantizes each 4-bit weight back to bfloat16, using the MLX scheme:
2. 2.the uint32 weight packs 8 × 4-bit values, little-endian (first value in the least significant bits);
3. 3.scale and bias are shared per group of group_size values (default 64);
4. 4.real_weight = q * scale + bias.
5. 5.Renames tensors, removing the language_model. prefix.
6. 6.Saves to incremental shards (~3 GB each), freeing memory after each one, and writes a model.safetensors.index.json for multi-shard loading.

The output is a standard safetensors model that llama.cpp converts without complaint.

Requirements

• —macOS on Apple Silicon (tested on M5, 24 GB RAM)
• —Python with torch and safetensors
• —llama.cpp (cloned from GitHub, for convert_hf_to_gguf.py and llama-quantize)
• —A fused MLX model directory (output of mlx_lm.fuse)

Usage

# 1. Convert the fused MLX model to a standard bf16 safetensors model
python3 mlx_to_gguf_ready.py \
    ~/path/to/fused-mlx-model \
    ~/path/to/output-dequant

# 2. Convert to GGUF (f16) with llama.cpp
python3 ~/llama.cpp/convert_hf_to_gguf.py \
    ~/path/to/output-dequant \
    --outfile ~/path/to/model-f16.gguf \
    --outtype f16

# 3. Quantize to Q4_K_M
llama-quantize \
    ~/path/to/model-f16.gguf \
    ~/path/to/model-Q4_K_M.gguf \
    Q4_K_M

Loading in Ollama (important)

The GGUF alone is not enough — without the correct chat template and a stop token, the model degenerates into loops and gibberish during long generations. Use a Modelfile like this (Mistral example):

FROM /path/to/model-Q4_K_M.gguf

TEMPLATE """{{ if .System }}[INST] {{ .System }}

{{ .Prompt }} [/INST]{{ else }}[INST] {{ .Prompt }} [/INST]{{ end }} {{ .Response }}</s>"""

SYSTEM """Your system prompt here."""

PARAMETER num_ctx 32768
PARAMETER temperature 0.8
PARAMETER stop "</s>"
PARAMETER stop "[INST]"

ollama create my-model -f Modelfile
ollama run my-model "Hello!"

The PARAMETER stop "</s>" line is the key fix for the loop/gibberish problem.

Notes and limitations

• —The dequantization reconstructs bfloat16 from already-quantized 4-bit weights, then llama-quantize re-quantizes to Q4KM. Two quantization steps in a row can introduce minor quality loss, but in practice it's negligible — the fine-tuned behavior is preserved.
• —Defaults assume bits=4, group_size=64 (the MLX default for Mistral/Mixtral). The script reads the actual values from config.json.
• —Tested on Mistral Small 3.1 24B. The same logic should apply to other MLX fine-tuned models, but verify the output generates coherent text before trusting it.

Validation / testing

The converted GGUF model was put through a test battery to confirm that the fine-tuned behavior survives the full MLX → dequant → GGUF → Q4KM chain and that there's no degeneration in long generations. All tests passed:

The converted model behaves identically to the original MLX model — the conversion is faithful. Any minor wording quirks come from the fine-tuning dataset, not from the conversion pipeline.

The long-generation test is the important one: before adding the [INST] template and PARAMETER stop "</s>" to the Modelfile, the model degenerated into loops and random (often non-Latin) tokens after a sentence or two. With those two lines, it's clean.

License

MIT — use it freely.