drivetechodyssey/Qwen-Drive-1.0-4B-VLM-mlx-4bit

Qwen-Drive-1.0-4B VLM · MLX 4-bit

The vision-language model inside Qwen/Qwen-Drive-1.0-4B, converted to MLX and quantized to 4 bits for Apple-silicon Macs. 2.8 GB, about 5.4 GB peak while running.

## Do not use this for question answering At 4 bits this model invents things that are not in the image. Asked for the one hazard that matters most, on three of sixteen demo images it reported a pedestrian crossing ahead; those three frames are a residential wall, trees and an empty road, with no crossing and no pedestrian. On a fourth it named an oncoming vehicle in a street of parked cars. None of its 48 answers matched the bf16 answer. The measurements are below. For reading a scene, use bf16 (8.5 GB, identical to the original weights) or 8-bit (4.8 GB).

What it is good for. Qwen-Drive's perception head and Planning Expert do not read the model's words. They read two intermediate tensors (the pre-merge vision patches and the image-token hidden states) and the full-attention K/V cache. Quantization noise that ruins a sentence is absorbed by those heads: on the bundled demo ground truth this 4-bit model scores the same as bf16 for 3D detection, occupancy, map segmentation and trajectory planning, while holding 2.8 GB instead of 8.5 GB. That is the only use this repository is published for.

Not for driving decisions. Research and demonstration only.

Use

Feature extraction for the perception head and the planner, through the MLX implementation in the `mlx-port` branch of the fork:

import mlx.core as mx
from qwen_drive_mlx import QwenDriveVLM, load_planner
from qwen_drive_perception_mlx import load_model, decode_boxes

vlm = QwenDriveVLM("drivetechodyssey/Qwen-Drive-1.0-4B-VLM-mlx-4bit")
heads = load_model("Qwen-Drive-1.0-4B/perception", dtype=mx.bfloat16)

vit, llm = vlm.perception_taps(input_ids, pixel_values, image_grid_thw, num_cams)
out = heads(mx.array(vit).astype(mx.bfloat16), mx.array(llm).astype(mx.bfloat16), img_metas)

It loads in mlx-vlm like the other two conversions, and will answer questions. Please read the warning above before you do that.

How it was made

1. 1.Extract the VLM. Every VLM tensor in the original model.safetensors starts with vlm.; the prefix is removed (723 tensors; lm_head is tied to the embeddings). config.json is the original vlm_config, with vision_config.model_type set to qwen3_5 because mlx-vlm 0.6.0 does not accept qwen3_5_vision. Tokenizer, processor and chat-template files are copied unchanged.
2. 2.Convert. mlx_vlm.convert --dtype bfloat16 -q --q-bits 4 --q-group-size 64 (affine).
3. 3.Restore float32 norms. The original keeps each linear-attention layer's A_log and norm.weight in float32 (48 tensors). The converter kept A_log but cast the 24 norm.weight tensors to bf16. Those 24 tensors are copied back from the original checkpoint. This fixes a drift in long bf16 answers; at 4 bits it changes nothing measurable, as the table below shows.

Checks

Measured on an M5 Max (64 GB) with mlx 0.31.2 and mlx-vlm 0.6.0, greedy decoding, nothing else using the GPU.

Question answering: fails

16 images from this repository's demo data (the current front, front-left and front-right frames of the four planning scenes, plus four front frames from the perception demo) x 3 questions: a long English scene description (400 tokens), a one-sentence hazard, and two or three sentences in Korean. Reference is the bf16 conversion, which reproduces the original weights exactly.

Restoring the float32 norms did not help here: before the restoration the same test gave 1 of 48 identical and 0.442 similarity, after it 0 of 48 and 0.437.

Perception and planning: same as bf16

The same six perception demo frames (6 to 8 cameras each) and four planning scenes, scored against the bundled ground truth. The VLM only supplies features here; the perception head and the planner run in MLX float32 and bf16 respectively, identical across the three columns.

Prompt processing is the same speed at every precision (1.9 to 2.0 k tok/s), because it is bound by arithmetic rather than weight bandwidth. The 4-bit model therefore does not make perception or planning faster; it makes them fit in less memory.

Everything above is one machine, one run, on this repository's demo data. There is no on-road validation.

License and citation

Apache License 2.0, the same as the original (see LICENSE). The weights are the Qwen team's (Alibaba Group); this repository only changes their format as described above.

@misc{zhou2026qwendrive10initialstepvisionlanguage,
      title={Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Driving},
      author={Xin Zhou and Zongchuang Zhao and Zhibo Yang and Mingsheng Li and Humen Zhong and Shuai Bai and Du Chu and Ruizhe Chen and Zhaohai Li and Jun Tang and Qiuyue Wang and Mingkun Yang and Jiazhao Zhang and Dayiheng Liu and Dingkang Liang and Xiang Bai},
      year={2026},
      eprint={2609.00111},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2609.00111},
}