ManniX-ITA/Qwen3.8-27B-Omnimerge-v6-MLX-VL-4bit

Qwen3.8-27B-Omnimerge-v6 — MLX 4-bit (Vision-Language)

Full multimodal 4-bit MLX quantization of `ManniX-ITA/Qwen3.8-27B-Omnimerge-v6`: text + image + video, runnable natively on Apple Silicon via `mlx-vlm`.

The base model is a task-arithmetic merge of three Qwen3.6 fine-tunes onto the Qwen3.8-27B base — same sources, weights and method as `Qwen3.6-27B-Omnimerge-v4`, moved to the newer base generation. Method, benchmark numbers and the merge forensics live on the base model card.

Inherited caveat — budget thinking headroom. v6 moves work out of the answer channel and into the reasoning channel (GPQA reasoning median 19,488 chars vs v4's 12,010). Six of 198 GPQA questions scored zero on an ~8192-token thinking budget alone. Set max_tokens generously.

Quantization

• —Type: MLX 4-bit (-q --q-bits 4 --q-group-size 64) via mlx_vlm.convert
• —Group size: 64
• —Effective bits/weight: 4.695 (reported by mlx_vlm.convert; identical to the v4 VL build — mlx-vlm keeps the vision tower at higher precision by default, so only the LM weights are 4-bit)
• —Shape on disk: 3 safetensors shards, 16.05 GB total
• —What is preserved: see the table below — measured from the produced model.safetensors.index.json, not assumed

Build environment

Converted on Linux (Ubuntu 24.04, glibc 2.39) on an RTX PRO 6000 Blackwell (sm_120), driver 610.43.02, using the CUDA MLX backend. End users on Apple Silicon run the native mlx runtime, which has no CUDA dependency.

mlx==0.30.0
mlx-cuda==0.30.0      <- ABI-coupled, must match mlx
mlx-lm==0.30.7        <- Qwen3.5/3.6/3.8 model_type support
mlx-vlm==0.3.12 (--no-deps)   <- last version that doesn't transitively bump mlx
transformers==5.16.1
torch==2.14.0+cpu     <- satisfies the Qwen3VL video processor's torchvision dep
                         without disturbing mlx-cuda's nvidia-cublas pin

mlx-vlm is installed with --no-deps precisely because its unpinned dependency set bumps mlx off 0.30.x and breaks the mlx / mlx-cuda ABI pairing. Its runtime imports (requests, soundfile, uvicorn, pydantic) are then installed explicitly — none of them can pull mlx.

Why this base converts on the 0.30.x pins at all: v6 is named for Qwen3.8 but still declares model_type: qwen3_5 / Qwen3_5ForConditionalGeneration, the same as v4. Checked before building: config keys 61 vs 61 with zero new fields (only transformers_version differs), and the tensor sets are identical at 1199 tensors each. mlx-lm==0.30.7's Qwen3.5 support therefore covers it — the "3.8" in the name is a base-generation label, not a new architecture.

Conversion recipe: `omnimergekit/scripts/mlx_convert.sh` (auto-detects vision_config and routes through mlx_vlm.convert). See `MLX_CONVERT.md` for the full pin rationale.

Verification status — read this before trusting the build

Everything below was measured on the produced artifact. Nothing here is inferred from the v4 build.

The MTP head is NOT in this build

The base carries 15 mtp.* tensors and the GGUF release deliberately retains them (as blk.64.*). mlx_vlm.convert drops them: the source index has 15 mtp.*, the output index has 0.

This is expected rather than a defect — mlx-vlm has no speculative-decoding path for this head — and MTP affects draft-acceptance rate, i.e. decode speed, not output quality. But do not assume parity with the GGUF tiers: if you came here expecting the MTP head, it is not here.

No inference smoke test was performed

This build has not been run. The conversion host is a CUDA box, and inference on the CUDA MLX backend fails on a cooperative_groups SDK mismatch — conversion works, generation does not. There is no Apple Silicon machine in this pipeline.

So the verification above is structural only: config, shard integrity, tensor survival and sidecar presence. It does not establish that the model generates coherent text, that the vision path works end to end, or that quality matches the base. The same was true of the v4 VL build. Treat first-run validation as yours to do, and please report anything broken.

Usage

pip install -U mlx-vlm

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config

repo = "ManniX-ITA/Qwen3.8-27B-Omnimerge-v6-MLX-VL-4bit"
model, processor = load(repo)
config = load_config(repo)

# Pure-text generation
prompt = apply_chat_template(processor, config,
    "Write a Rust function that returns the n-th Fibonacci number iteratively.")
print(generate(model, processor, prompt, max_tokens=4096, verbose=True))

# Vision (with an image)
prompt = apply_chat_template(processor, config,
    "Describe the image in detail, then state what's likely happening.",
    num_images=1)
print(generate(model, processor, prompt,
    max_tokens=4096, verbose=True, image=["path/to/image.png"]))

# Video (with a clip)
prompt = apply_chat_template(processor, config,
    "Summarize what happens in this video.", num_videos=1)
print(generate(model, processor, prompt,
    max_tokens=4096, verbose=True, video=["path/to/clip.mp4"]))

The base model emits Qwen3.5-family reasoning tags (<think>...</think>). Strip them in post-processing or use a chat template wrapper that handles them.

Memory & speed

• —Resident memory: ~17-18 GB expected (M-series, 32 GB+ recommended) — extrapolated from the equally-sized v4 VL build, not measured on this build
• —Context length: inherits the base model's context (RAM permitting)
• —Speed is comparable to other Qwen3-VL 27B 4-bit MLX builds; depends on chip generation

Related

• —Base merge: `ManniX-ITA/Qwen3.8-27B-Omnimerge-v6`
• —GGUF (19 tiers, MTP retained): `ManniX-ITA/Qwen3.8-27B-Omnimerge-v6-MTP-GGUF`
• —Ollama tags: `mannix/omnimerge-v6` (needs ollama >= 0.32.12)
• —v4 MLX VL 4-bit (previous generation): `ManniX-ITA/Qwen3.6-27B-Omnimerge-v4-MLX-VL-4bit`
• —Methodology + scripts: `mann1x/omnimergekit`

License

Apache 2.0 — inherits from the Qwen3.8 base. See the base model card for the full attribution list (Qwen team, rico03, ValiantLabs, kai-os).