Modelpublic

m1rkocasu/Apertus-v1.5-8B-text-MLX-4bit-DWQ

sourceHugging Faceapache-2.0updated 8d agoView on Hugging Face

1likes446downloads

Model Card

<h1 align="center">Apertus v1.5 8B text — MLX 4-bit DWQ</h1>

<p align="center"> <a href="https://huggingface.co/swiss-ai/Apertus-v1.5-8B">Base model</a> | <a href="https://apertus-ai.org/">Apertus</a> | <a href="https://github.com/ml-explore/mlx-lm">mlx-lm</a> | <a href="https://github.com/jundot/omlx">oMLX</a> | <a href="#the-family">The family</a> <br> <b>Format</b>: MLX | <b>Weights</b>: 4-bit DWQ, 4.53 GB | <b>License</b>: <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache 2.0</a> </p>

This repository holds the text branch of Apertus 1.5 8B, converted to MLX and quantized to 4-bit DWQ. It runs on Apple silicon through mlx-lm.

Apertus 1.5 is the fully open model of the Swiss AI Initiative, built at EPFL, ETH Zurich and the Swiss National Supercomputing Centre on open data.

This conversion was made independently of the Apertus release.

Model summary

—The model size the sidebar of this page reports is smaller than 8.05B. That figure counts the elements of the stored tensors, and MLX packs quantized weights into 32-bit containers. The parameter count of the model is the one in the table. </p>

The family

Six builds of the same text branch, measured on the same corpus with the same script. Perplexity is measured on the test split of Salesforce/wikitext, configuration wikitext-2-raw-v1, over 200 non-overlapping windows of 512 tokens, which is 102,400 scored tokens.

<div align="center" style="overflow-x:auto"> <table style="border-collapse:collapse;margin:0 auto"> <thead><tr><th style="padding:9px 14px;border-bottom:2px solid #45BED9;color:#45BED9;font-weight:600;font-size:13px;text-align:left">Build</th><th style="padding:9px 14px;border-bottom:2px solid #45BED9;color:#45BED9;font-weight:600;font-size:13px;text-align:center">Bits per weight</th><th style="padding:9px 14px;border-bottom:2px solid #45BED9;color:#45BED9;font-weight:600;font-size:13px;text-align:center">Size</th><th style="padding:9px 14px;border-bottom:2px solid #45BED9;color:#45BED9;font-weight:600;font-size:13px;text-align:center">Perplexity</th><th style="padding:9px 14px;border-bottom:2px solid #45BED9;color:#45BED9;font-weight:600;font-size:13px;text-align:center">Against the 8-bit</th><th style="padding:9px 14px;border-bottom:2px solid #45BED9;color:#45BED9;font-weight:600;font-size:13px;text-align:left">Use it when</th></tr></thead> <tbody> <tr><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:left"><a href="https://huggingface.co/m1rkocasu/Apertus-v1.5-8B-text-MLX-8bit">8-bit</a></td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">8.500</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">8.54 GB</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">10.4088</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">reference</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:left">quality first, and the reference every other build here is measured against.</td></tr> <tr><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:left"><a href="https://huggingface.co/m1rkocasu/Apertus-v1.5-8B-text-MLX-mxfp8">MXFP8</a></td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">8.250</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">8.31 GB</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">10.4748</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">+0.6%</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:left">you want the floating-point format. In perplexity it sits next to the 6-bit, which is smaller.</td></tr> <tr><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:left"><a href="https://huggingface.co/m1rkocasu/Apertus-v1.5-8B-text-MLX-6bit">6-bit</a></td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">6.500</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">6.54 GB</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">10.4618</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">+0.5%</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:left">two GB less than the 8-bit for half a percent of perplexity.</td></tr> <tr><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:left"><a href="https://huggingface.co/m1rkocasu/Apertus-v1.5-8B-text-MLX-5bit">5-bit</a></td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">5.500</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">5.54 GB</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">10.5261</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">+1.1%</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:left">three GB less than the 8-bit, and still within one and a half percent of it.</td></tr> <tr><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:left"><strong>4-bit DWQ</strong> (this repository)</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">4.500</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">4.53 GB</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">10.9691</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">+5.4%</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:left">the smallest build that stays close: distillation buys back most of what plain 4-bit gives away, at the same file size.</td></tr> <tr><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:left"><a href="https://huggingface.co/m1rkocasu/Apertus-v1.5-8B-text-MLX-mxfp4">MXFP4</a></td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">4.250</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">4.28 GB</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">10.8522</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">+4.3%</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:left">the smallest build here, and now calibrated: closer to the 8-bit than the 4-bit DWQ, a quarter of a GB lighter.</td></tr> </tbody> </table> </div>

—The 8-bit build is the reference because a bfloat16 run does not fit the 24 GB machine these were made on: its 15 GB of weights page to disk. The step from 8 bits to 6 costs half a percent, which makes a large gap between bfloat16 and 8 bits unlikely. That is an inference, not a measurement.<br>
—Perplexity compares quantizations of one model on one corpus. It says nothing about how this model compares to a different model, and nothing about how well it follows instructions. </p>

What was left out

These variants were measured on the same corpus and are not published. The table is here so that nobody repeats the work.

<div align="center" style="overflow-x:auto"> <table style="border-collapse:collapse;margin:0 auto"> <thead><tr><th style="padding:9px 14px;border-bottom:2px solid #45BED9;color:#45BED9;font-weight:600;font-size:13px;text-align:left">Variant</th><th style="padding:9px 14px;border-bottom:2px solid #45BED9;color:#45BED9;font-weight:600;font-size:13px;text-align:center">Bits per weight</th><th style="padding:9px 14px;border-bottom:2px solid #45BED9;color:#45BED9;font-weight:600;font-size:13px;text-align:center">Size</th><th style="padding:9px 14px;border-bottom:2px solid #45BED9;color:#45BED9;font-weight:600;font-size:13px;text-align:center">Perplexity</th><th style="padding:9px 14px;border-bottom:2px solid #45BED9;color:#45BED9;font-weight:600;font-size:13px;text-align:left">Why it is not published</th></tr></thead> <tbody> <tr><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:left">MXFP4, rounded to nearest (published here until 2026-09-19)</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">4.250</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">4.28 GB</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">11.71</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:left">replaced by the calibrated MXFP4, same size</td></tr> <tr><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:left">mixed 4/6, affine</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">5.000</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">5.03 GB</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">11.24</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:left">bigger than the 4-bit DWQ build, and weaker</td></tr> <tr><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:left">4-bit, affine, no distillation</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">4.500</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">4.53 GB</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">11.59</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:left">same size as the 4-bit DWQ build, and weaker</td></tr> <tr><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:left">3-bit, group size 32</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">4.000</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">4.03 GB</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">66.06</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:left">unusable</td></tr> <tr><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:left">3-bit DWQ, group size 64</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">3.500</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">3.52 GB</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">50.91</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:left">unusable</td></tr> <tr><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:left">mixed 3/6, affine</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">4.250</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">4.28 GB</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">140.93</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:left">unusable</td></tr> <tr><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:left">3-bit, group size 64</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">3.500</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">3.52 GB</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:center">169.79</td><td style="padding:8px 14px;border-bottom:1px solid rgba(128,128,128,0.25);font-size:14px;text-align:left">unusable</td></tr> </tbody> </table> </div>

Three bits break this model. At a group size of 64 it answers fluently and wrongly, completing "The capital of Switzerland is" with "not a good idea"; at 32 it answers "Bern" and still reaches a perplexity of 66. Distillation recovers a large share of the loss, from 169.79 to 50.91, and the result is still five times the reference.

The two four-bit rows are here for a different reason: they work, but each is beaten by a published build that is the same size or smaller. Nothing between 4.25 and 4.5 bits per weight was worth publishing besides the two that are.

<div style="border-left:4px solid #45BED9;background:rgba(69,190,217,0.07);border-radius:0 4px 4px 0;padding:12px 16px;margin:14px 0"> <p style="margin:0"><strong>Need images or audio?</strong> The whole model, this same decoder plus the vision and audio towers in float32, is a separate family, converted to MLX for mlx-vlm. Start at <a href="https://huggingface.co/m1rkocasu/Apertus-v1.5-8B-MLX-8bit">Apertus v1.5 8B MLX</a>.</p> </div>

This build

Every quantizable layer holds 4 bits with a group size of 64, and the scales and biases were then trained by distillation from the 8-bit build. The integers are the ones plain 4-bit quantization produces. What changed is the pair of continuous numbers that maps each group of 64 integers back to real values, which moves the perplexity from 11.5926 to 10.9691 at exactly the same file size.

Its perplexity is 10.9691, +5.4% against the 8-bit build, on the protocol described above.

bash

mlx_lm.convert --hf-path <text branch> \
  --mlx-path Apertus-v1.5-8B-text-MLX-4bit -q --q-bits 4 --q-group-size 64

The work ran with mlx-lm 0.31.3 and mlx 0.32.0 on macOS 26.5.

Run it

bash

pip install mlx-lm
mlx_lm.generate --model m1rkocasu/Apertus-v1.5-8B-text-MLX-4bit-DWQ \
  --prompt "Name the capital of Switzerland and say one sentence about it." \
  --max-tokens 200

From Python:

python

from mlx_lm import load, generate

model, tokenizer = load("m1rkocasu/Apertus-v1.5-8B-text-MLX-4bit-DWQ")
messages = [{"role": "user", "content": "Name the capital of Switzerland."}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
print(generate(model, tokenizer, prompt=prompt, max_tokens=200))

Behind an OpenAI-compatible endpoint:

bash

mlx_lm.server --model m1rkocasu/Apertus-v1.5-8B-text-MLX-4bit-DWQ

In oMLX, search for m1rkocasu/Apertus-v1.5-8B-text-MLX-4bit-DWQ and download it. Deliberation is read natively: the model opens it with <think> and closes it with </think>, which is what oMLX looks for.

What the text branch is

The upstream release is multimodal. It reads images and audio as well as text, and its architecture is Apertus1p5ForConditionalGeneration, which neither mlx-lm 0.31.3 nor mlx-vlm 0.6.17 implements.

This repository holds the text decoder on its own, declared as ApertusForCausalLM so that mlx-lm loads it. The token embedding goes from 266752 rows to 131072. The rows that go are the image and audio codebooks, which start at index 131272 and 262344 in the upstream vocabulary. The upstream output_vocab_size is already 131072, so the language modelling head is untouched and no text token is lost.

The model reads and writes text. It does not accept images or audio. The whole model, towers included, is a different family: Apertus v1.5 8B MLX.

Limitations

The quantization inherits every limitation of the upstream model, which its model card describes. No output filter ships with these weights.

Quality here is measured by perplexity on one English corpus. Apertus 1.5 is multilingual, and the effect of quantization on languages other than English is not measured in this collection. Instruction following, reasoning and tool use are also unmeasured.

License and acceptable use

The weights stay under the Apache 2.0 license of the upstream release. Use is also subject to the Apertus 1.5 acceptable use policy and privacy policy:

For removal of personal or copyrighted data, write to the Swiss AI Initiative at <llm-privacy-requests@swiss-ai.org> or <llm-copyright-requests@swiss-ai.org>.

Credits

The model is the work of the Swiss AI Initiative. This repository adds the MLX conversion, the quantization and the measurements above.

bibtex

@misc{ApertusV15,
  author       = {{Swiss AI Initiative}},
  title        = {Apertus v1.5},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/swiss-ai/Apertus-v1.5-8B}},
  note         = {EPFL, ETH Zurich, and the Swiss National Supercomputing Centre}
}