Modelpublic

Arm/qwen3-vl-2b-instruct-q4-k-m-ggml-llama-cpp-vivo-x300

sourceHugging Faceapache-2.0updated 20d agoView on Hugging Face

0likes344downloads

Model Card

Qwen3-VL-2B-Instruct optimized for Arm-based mobile CPUs with SME2

Qwen3-VL-2B-Instruct vision-language generation, with the text decoder quantized to Q4KM and the vision projector kept at Q8_0, exported as a GGUF .gguf pair for the llama.cpp runtime on Arm-based mobile CPUs with SME2.

Summary

This repository contains an Arm-optimized version of Qwen/Qwen3-VL-2B-Instruct for image-and-text to text generation. The model is provided as GGUF .gguf files for the llama.cpp runtime, targeting Mobile CPU systems.

A vision-language model is consumed by llama.cpp as a pair of files, and both are shipped here: the text decoder, quantized to Q4KM — a mixed scheme, not a uniform 4-bit one: most weight tensors are INT4 (Q4K), the more sensitive ones INT6 (Q6K), activations are dynamically quantized to INT8 at runtime, and the normalization tensors stay in FP32 — and an mmproj sibling holding the vision encoder and projector, kept at Q8_0. The projector never passes through llama-quantize, and the importance matrix used for the text decoder is not applied to it — an importance matrix collected on the decoder describes nothing about vision-tower activations. The text GGUF alone cannot see images; the two files must be loaded together.

This version is intended to demonstrate efficient inference on Arm-based platforms while preserving the original model's intended behavior. Arm evaluated this model on MMLU-Redux 2.0 and measured performance on a representative evaluation target.

Key results

Area	Result
Model format	GGUF `.gguf` (llama.cpp), text decoder plus mmproj projector
Target device class	Mobile CPU
Reference device	vivo X300 (C1-Ultra, C1-Premium, C1-Pro; Android 16 / OriginOS 6)
Primary performance result	30.87 tokens/sec, 3239.43 ms TTFT
Accuracy result	MMLU 65.05% (5-shot)
Size / memory result	1056.11 MB, 3.11x smaller than the f16 baseline

Original model

Field	Value
Original model	Qwen/Qwen3-VL-2B-Instruct
Original source	Hugging Face
Original developer	Alibaba Cloud (Qwen team)
Original model card	Qwen/Qwen3-VL-2B-Instruct
Original license	Apache-2.0

Model files

File	Description
`Qwen__Qwen3-VL-2B-Instruct_llamacpp_optimized.gguf`	Arm-optimized Q4KM text decoder for deployment
`Qwen__Qwen3-VL-2B-Instruct_llamacpp_optimized_mmproj.gguf`	Q8_0 vision encoder and projector, loaded alongside the decoder
`example.py`	Minimal inference example
`pyproject.toml`	Pinned runtime dependencies for `example.py`, resolved with uv
`uv.lock`	Locked dependency resolution for `pyproject.toml`
`.python-version`	Python version the example was validated against
`config.yaml`	Model I/O contract used by the example
`sample_input.jpg`	Example input image
`predictions.json`	Example output produced by the optimized model
`benchmarks/`	f16 baseline and Arm-optimized benchmark records

Performance

Performance was measured on the reference configuration below. Results are intended to make the optimization reproducible but do not guarantee identical performance on every Arm-based system.

Reference configuration

Field	Value
Device / platform	vivo X300
CPU / accelerator	C1-Ultra, C1-Premium, C1-Pro (8 cores), cpu; 4 threads
OS	Android 16 / OriginOS 6
Runtime	llama.cpp
Backend / delegate	cpu (ggml), built with -march=armv8.6-a+dotprod+i8mm
Batch size	1
Precision	Q4KM mixed K-quant weights on the text decoder: INT4 (Q4K) for most tensors and INT6 (Q6K) for the more sensitive ones, asymmetric, static, per-group, with activations dynamically quantized to INT8 at runtime; normalization tensors in FP32; vision encoder and projector in Q8_0
Runs	30 measured

Performance results

Metric	Original / baseline	Arm-optimized	Improvement
Throughput	10.45 tokens/sec	30.87 tokens/sec	2.95x
TTFT	23725.45 ms	3239.43 ms	7.32x
p50 latency	36064.06 ms	7387.75 ms	4.88x
Model size	3287.65 MB	1056.11 MB	3.11x smaller
Peak memory	3458.97 MB	2269.35 MB	1.52x less

The benched shape is a 512-token prompt with 128 generated tokens.

Accuracy

Accuracy was evaluated using the same preprocessing, input resolution, and evaluation protocol described below. Where possible, the optimized model is compared against the original model under the same evaluation conditions.

Evaluation setup

Field	Value
Dataset	MMLU-Redux 2.0
Split	test, 5-shot
Number of samples	5330
Metric(s)	MMLU accuracy
Evaluation runtime	llama.cpp

Scoring follows the lettered multiple-choice formulation that published MMLU-family numbers use: subject header, lettered options, prompt ending in "Answer:", and only the next-token logits for the option letters are scored. Same-subject exemplars are prepended from MMLU's own dev split, and no chat template is applied.

Accuracy results

Metric	Original / baseline	Arm-optimized	Change
MMLU accuracy (5-shot)	65.42%	65.05%	-0.37 pp

MMLU-Redux is a text benchmark. It measures the quantized decoder, which is the only half of the pair this recipe touches, but it says nothing about image grounding. Users who care about the vision path should evaluate it separately.

Accuracy was measured using the evaluation setup described above. Users should re-evaluate the model on their own data before production use.

Arm optimization approach

Arm optimized this model for efficient inference on Arm-based platforms using a hardware-aware conversion and validation flow.

For this release, Arm used:

Optimization area	Applied?	Notes
Model conversion	Yes	Converted to GGUF `.gguf` for llama.cpp; the vision encoder and projector are emitted as a separate mmproj file
Quantization	Yes	Mixed K-quant Q4KM weights on the text decoder — INT4 (Q4K) for most tensors, INT6 (Q6K) for the more sensitive ones such as attnv and ffndown, asymmetric, static, per-group, with INT8 dynamic activation quantization at runtime — calibrated with an importance matrix over 200 randomly selected WikiText samples. The vision encoder and projector are converted straight to Q8_0 and never see llama-quantize or the importance matrix
Runtime/backend selection	Yes	llama.cpp CPU backend (ggml), built with -march=armv8.6-a+dotprod+i8mm
Graph/runtime compatibility updates	Yes	Performed as part of the llama.cpp export pipeline
Accuracy validation	Yes	Compared against the original model or published baseline
Performance validation	Yes	Measured on the reference Arm platform

The goal of this process is to improve deployment characteristics such as latency, memory use, model size, and runtime compatibility while preserving the model's intended behavior. Detailed conversion scripts, calibration configuration, or backend-specific implementation details may be provided separately where appropriate.

Using this model

Install dependencies

Dependencies are declared in pyproject.toml, which ships with this repository. Resolve and install them into a local virtual environment with uv:

bash

uv python install
uv sync --frozen

llama-cpp-python publishes no wheels on PyPI, so this compiles the vendored llama.cpp from source on first install; cmake and a C++ compiler must be present.

Download the model

Download both GGUF files from this repository and place them next to example.py. The text decoder alone cannot see images, so the example checks for both up front and fails with the missing filename rather than running half the pair.

Run the example

bash

uv run example.py

Note: The Python/uv example runs on AWS Graviton (Ubuntu arm64) to confirm runtime compatibility only, and is intended as a guideline for building an equivalent run script on Mobile CPU systems.

The same pair can be served over an OpenAI-compatible endpoint, using the llama-server binary from a llama.cpp build:

bash

llama-server \
  -m Qwen__Qwen3-VL-2B-Instruct_llamacpp_optimized.gguf \
  --mmproj Qwen__Qwen3-VL-2B-Instruct_llamacpp_optimized_mmproj.gguf \
  --jinja -c 4096

The --jinja flag is load-bearing: without it the server falls back to a generic chat template and the model never sees its own image and control tokens.

Expected input

Property	Value
Input shape	[1, 4096] input_ids (tokenized prompt); image inputs are injected by libmtmd as vision embeddings at the image placeholder positions
Input type	int32 token ids
Input range	vocabulary ids, 0 to 151935
Preprocessing	Apply the chat template embedded in the `.gguf`, then tokenize with the embedded qwen2-style BPE tokenizer (vocabsize 151936, bos token id 151643), truncating at nctx 4096. Images are normalized with mean 0.5 and std 0.5 at image size 768, patch size 16 and spatial merge size 2 by the mmproj projector, and merged into the token stream by libmtmd

Expected output

Property	Value
Output shape	[1, seq_len, 151936]
Output type	per-token logits over the vocabulary; last-position logits only unless all logits are requested
Postprocessing	Greedy (argmax) decoding was used for reproducible evaluation; the GGUF records topk 20, topp 0.8 and temperature 0.7 as the vendor sampling defaults. The KV cache is internal to llama.cpp — feed the sampled token back one at a time. Detokenize with the embedded BPE tokenizer; stop on eos token id 151645 or when maxnewtokens is reached

Intended use

This model is intended for developers evaluating image-text-to-text workloads on Arm-based platforms. It is suitable as a reference implementation for benchmarking, prototyping, and integration exploration.

Limitations

—Performance depends on the target device, runtime version, backend/delegate support, memory configuration, and system load.
—Accuracy was evaluated on MMLU-Redux 2.0 test (text, 5-shot) and may not generalize to all domains, and does not cover image grounding at all.
—This release preserves the original model's intended task and behavior, but users should validate it for their own application, data, and deployment environment.
—This repository is not a replacement for the original model documentation.

Additional notes

—The pair must stay together. The mmproj sibling, Qwen__Qwen3-VL-2B-Instruct_llamacpp_optimized_mmproj.gguf, is published alongside the text decoder; loading the decoder without it yields a text-only model that silently ignores images rather than failing. libmtmd, which loads the pair, ships in the llama-mtmd-cli and llama-server binaries from a llama.cpp build.
—The on-device figures cover the text decoder only: upstream llama-bench has no projector support, so the vision tower never reached the phone. There is no image-encoding cost in them, and peak memory understates real multimodal inference by the projector's residency plus vision KV.
—The baseline is f16, not bf16. ggml has no aarch64 bf16 kernel, so a bf16 reference would be measured through a scalar fallback and would inflate every speedup ratio. Re-encoding the bf16 checkpoint to f16 is exact for every weight in f16's normal range and loses precision only on weights small enough to fall into f16 subnormals.
—The model records a context length of 262144; the evaluated configuration used a 4096-token window.
—Sample input: sample_input.jpg is derived from Living room (Unsplash).jpg) by Jarosław Ceborski, via Wikimedia Commons (CC0 1.0).

About this version

Original Model: Qwen/Qwen3-VL-2B-Instruct by Alibaba Cloud (Qwen team) - Repository

Optimization/conversion: Arm-Optimized version for execution on Arm-based platforms.

Converted/optimized by: Arm

License: The Original Model and the Optimized Model are subject to Apache-2.0.

This repository contains a converted or optimized version of the Original Model (the “Optimized Model”). The Original Model has been converted or optimized as described above for execution on Arm-based platforms.

No retraining or fine-tuning of the Original Model was performed as part of the conversion or optimization. The conversion or optimization was not intended to change the Original Model’s behavior or intended use.

Original Model and Documentation

For information about the Original Model, including its development, training data, intended uses, limitations and other relevant information, please refer to the Original Model repository. Information in that repository was provided by the original developer or other third parties and, unless expressly stated otherwise, has not been independently verified by Arm.

Licenses and Third-Party Terms

Use of the Original Model and the Optimized Model is subject to the applicable licenses, usage restrictions and other terms identified above and in the relevant repositories. Publication of the Optimized Model does not grant any rights beyond those provided under the applicable license terms.

You are responsible for reviewing those terms and ensuring that your use of the Original Model and the Optimized Model is permitted.

Purpose of this Release

The Optimized Model is provided as a reference implementation to demonstrate and evaluate execution and performance on Arm-based systems. It is not a production-ready or supported solution.

Arm’s publication of the Optimized Model does not constitute an endorsement or certification of the Original Model or a representation that the Optimized Model is suitable for production use or any particular purpose.

To the fullest extent permitted by applicable law (i) the Optimized Model is provided “as is.” Arm makes no representations or warranties that the Original Model, the Optimized Model or their outputs are accurate, safe, secure, non-infringing, legally compliant, suitable for production use or fit for any particular purpose; and (ii) Arm will not be liable for any loss or damage arising from or in connection with the Optimized Model, its use or its outputs.

You are responsible for independently evaluating the Optimized Model, its outputs and its suitability for your intended use, including compliance with applicable legal, regulatory, safety and security requirements.

Arm does not commit to provide ongoing support, maintenance or updates for the Optimized Model. Any use of or reliance on the Optimized Model or its outputs is at your own risk.