jason-schulz/KAT-Coder-V2.5-Dev-VL-oQ4e-mtp

KAT-Coder-V2.5-Dev-VL-oQ4e-mtp

`Kwaipilot/KAT-Coder-V2.5-Dev` with its vision tower and MTP head restored from the base model it was fine-tuned from, quantized to 4-bit for Apple Silicon. 20 GB.

This is the 4-bit member of a set built from one bf16 master:

Read the accuracy column as a tie, not a ranking. All three were measured the same day, same settings (full 164 problems, batch_size=1, thinking enabled, deterministic profile). The oQ8e build scored 146/164 on an earlier run and 141/164 on the run in this table — a 5-problem swing on an unchanged model, which is as large as the entire spread between levels. Speculative decoding plus Metal batching numerics means "deterministic" is not bit-reproducible here. The honest conclusion is that 8, 6, and 4-bit are indistinguishable on this benchmark, not that 4-bit is better.

Read this first: what you need to load it. The oQ4e quantization is not a proprietary format. oQ writes standard mlx-lm safetensors with a standard per-module quantization config, and portable oQ models load in stock MLX apps. *What makes this repo oMLX-only is the restored MTP head. Stock `mlx-vlm` does a strict load and rejects the 42 extra `language_model.mtp. tensors: ValueError: Received 42 parameters not in model`. So as published this needs oMLX on Apple Silicon — for the MTP tensors, not the quant. Not loadable by llama.cpp, vLLM, or Transformers (MLX safetensors either way).

What was done

Kwaipilot's open-weight release ships language-model weights only — the vision components and the multi-token-prediction head were stripped. Both are recoverable because KAT's config.json is byte-identical to Qwen/Qwen3.6-35B-A3B's except for one field, and the text architecture is unchanged.

The tensor key set is identical to an oMLX oQ8e-mtp build of the stock base model — 2052 tensors, with the MTP head correctly switch_mlp-named.

Verification

Measured on this exact checkpoint:

• —Loads as a VLM. Server log reports VLMBatchedEngine loaded, not the silent VLM to LLM fallback that yields a vision-less model.
• —Vision. 8-part probe (dense code screenshots, high-entropy strings, UI defect-spotting, mixed heading sizes, sparse-glyph sweeps). Transcription is exact except for two failure modes the native Qwen3.6 base reproduces as well: the capital Q in sk-7Qv3... reads as lowercase, and a sparse 384x320 @64px image misreads. No regression against the oQ8e build.
• —MTP. 79.3% draft acceptance, 2.54 tokens/cycle over 8 generations.
• —HumanEval. 0.8963 pass@1 (147/164), 722.5s, thinking enabled.

Caveat on MTP at low bit depths. oMLX warns that quantizing a preserved MTP head can collapse draft acceptance toward ~0%. That did not happen here, and the likely reason is that the imatrix was recomputed with the MTP head present rather than reused from a pre-graft cache. If you build your own level from a stale imatrix, check the MTP[n] ... accept= line before trusting the speedup — do not assume 4-bit is universally safe for MTP-preserved heads.

Reproducing

Build scripts live in the oQ8e repo: `build/merge_kat_vl.py` and `build/graft_mtp.py`, with full instructions in its README. Budget ~146 GB of free disk; peak RAM is about 8 GB.

Steps 1 and 2 produce the bf16 master. This repo is step 3 with "oq_level": 4:

curl -X POST http://127.0.0.1:8899/admin/api/oq/start \
  -H 'Content-Type: application/json' -d '{
    "model_path": "'"$HOME"'/Models/_build/KAT-Coder-V2.5-Dev-VL",
    "oq_level": 4, "enhanced": true, "dtype": "bfloat16", "group_size": 64,
    "preserve_mtp": true }'

Then set mtp_enabled on the new model id — it is per-model and off by default, and a model with MTP disabled logs no acceptance line at all, which is easy to misread as a collapsed head.

Limitations

• —*oMLX only as published** — because of the MTP head, not the quantization. Strip the `mtp. tensors and set textconfig.mtpnumhiddenlayers = 0` and the remainder is a portable oQ checkpoint that loads in stock MLX apps; you lose speculative decoding, keep vision and the text model.
• —Vision is untrained for this checkpoint. Good for screenshots of code/terminals/UI; unreliable for isolated large glyphs and occasionally for single characters in random strings.
• —Set a `max_tokens` cap. Long reasoning excursions stall rather than error in an agent loop.
• —HumanEval is a narrow benchmark. "Indistinguishable on HumanEval" is not "indistinguishable in agentic coding use" — if you have a workload that discriminates between these levels, that result is more informative than the table above.
• —Chinese/English only, inherited from the base.

Credits

• —Kwaipilot — KAT-Coder-V2.5-Dev
• —Qwen — Qwen3.6-35B-A3B, source of the vision tower and MTP head
• —oMLX — quantization and serving
• —The KAT-Coder community discussions, where the vision and MTP transplants were first proposed and tested

Apache-2.0, inherited from both parents.