Modelpublic

popfido/Qwythos-9B-Claude-Mythos-5-1M-Q8-MLX-MTP-Visual

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face

0likes28downloads

Model Card

Qwythos-9B-Claude-Mythos-5-1M — Q8 (8-bit) MLX (MTP + Vision)

True 8-bit MLX build of [empero-ai/Qwythos-9B-Claude-Mythos-5-1M-GGUF](https://huggingface.co/empero-ai/Qwythos-9B-Claude-Mythos-5-1M-GGUF) (Qwen3.5-9B, hybrid GatedDeltaNet + attention) with the MTP head and the Qwen3.5-9B vision tower.

Pipeline: the repo's `MTP-Q8_0` GGUF → float16 MLX (via [popfido/gguf2mlx](https://github.com/popfido/gguf2mlx/tree/feat/qwen35-mtp-vision), merged with the F16 mmproj) → oQ level 8 (8-bit) quantization. ~10 GB — the middle tier between oQ4 (4-bit, ~6 GB) and BF16 (full-precision f16, ~19 GB).

Note: an earlier revision of this repo stored f16 weights dequantized from the Q8 GGUF (same 19 GB as BF16, but Q8 quality — redundant). It has been replaced with this genuine 8-bit quantization.

Quantization

—Q8 — uniform 8-bit, affine, group size 64 (oMLX Universal Dynamic Quantization). 8-bit is effectively lossless here: greedy generation matches the full-precision model (Paris / counting / arithmetic all correct, no drift).
—MTP preserved (preserve_mtp=True): mtp.* tensors and mtp_num_hidden_layers kept so the Native MTP toggle works post-quantization.

The underlying float16 conversion was verified bit-exact vs llama.cpp (text) and the vision tower matches llama.cpp mtmd output — see the BF16 repo card for the full verification + Qwen3.5 conversion notes (zero-centered RMSNorm, A_log = log(−ssm_a), strided→grouped value-heads, YaRN).

Usage

python

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template

model, processor = load("popfido/Qwythos-9B-Claude-Mythos-5-1M-Q8-MLX-MTP-Visual")
prompt = apply_chat_template(processor, model.config, "Describe this image.", num_images=1)
print(generate(model, processor, prompt, image=["image.png"], max_tokens=128).text)

Recommended sampling: temperature 0.6, top_p 0.95, top_k 20, repetition_penalty 1.05.

License & attribution

Inherits apache-2.0 from the base model. All credit for the model to empero-ai; this repo is an MLX-format conversion + oQ8 quantization only.