Modelpublic

Arm/qwen3-5-2b-q4-k-m-ggml-llama-cpp-vivo-x300

sourceHugging Faceapache-2.0updated 19d agoView on Hugging Face

0likes904downloads

Model Card

Qwen3.5-2B optimized for Arm-based mobile CPUs with SME2

Qwen3.5-2B text generation, quantized to Q4KM GGUF for the llama.cpp runtime on Arm-based mobile CPUs with SME2.

Summary

This repository contains an Arm-optimized version of Qwen/Qwen3.5-2B for text generation. The model is provided in GGUF format for the llama.cpp runtime, targeting Mobile CPU systems.

This version is intended to demonstrate efficient inference on Arm-based platforms while preserving the original model's intended behavior. Arm has evaluated this model on edinburgh-dawg/mmlu-redux-2.0 and measured performance on a representative evaluation target.

The shipped artifact is the Q4KM language-model GGUF. A separate f16 multimodal projector GGUF is included in the bundle but is not quantized, benchmarked, or evaluated, so every number on this card describes the text decoder only.

Key results

Area	Result
Model format	GGUF (llama.cpp), Q4KM k-quant
Target device class	Mobile CPU
Reference device	vivo X300 (C1-Ultra, C1-Premium, C1-Pro, Android 16 / OriginOS 6)
Primary performance result	22.09 tok/s decode throughput, 983.29 ms time to first token
Accuracy result	MMLU 62.4578% (0-shot)
Size / memory result	1215.36 MB, 2.96 x smaller than the f16 baseline

Original model

Field	Value
Original model	Qwen/Qwen3.5-2B
Original source	Hugging Face
Original developer	Alibaba Cloud (Qwen team)
Original model card	https://huggingface.co/Qwen/Qwen3.5-2B
Original license	apache-2.0

Model files

File	Description
`Qwen__Qwen3.5-2B_llamacpp_optimized.gguf`	Q4KM quantized language model, 1215 MB
`Qwen__Qwen3.5-2B_llamacpp_original.gguf`	f16 reference language model, 3600 MB
`mmproj-Qwen__Qwen3.5-2B_llamacpp_optimized.gguf`	f16 vision projector, 637 MB, unquantized
`example.py`	Runnable inference example; writes sample_input.txt and predictions.json
`config.yaml`	Input/output shapes, preprocessing, quantization and runtime details
`metadata.yaml`	Machine-readable metadata for the Arm-optimized model
`pyproject.toml`	Python dependency spec for the example
`uv.lock`	Locked dependency versions, resolved on the aarch64 host below
`.python-version`	Pinned Python release used by uv
`sample_input.txt`	The prompt the committed prediction was generated from
`predictions.json`	Known-good output of `example.py`
`benchmarks/`	Benchmark records for the f16 baseline and the Q4KM model

The GGUF carries its own vocab, merges and chat template, so no external tokenizer files are needed.

Performance

Performance was measured on the reference configuration below. Results are intended to make the optimization reproducible but do not guarantee identical performance on every Arm-based system.

Reference configuration

Field	Value
Device / platform	vivo X300
CPU / accelerator	C1-Ultra, C1-Premium, C1-Pro / cpu
OS	Android 16 / OriginOS 6
Runtime	llama.cpp
Backend / delegate	ggml
Batch size	1
Precision	q4km, K-quant
Runs	5 warm-up, 20 measured
Threads	4
Prompt / generation length (tokens)	128 / 128

Measurement conditions. Each measured run consumes a 128-token prompt and generates 128 tokens, using 4 CPU threads, 5 warm-up runs and 20 measured runs. Each run starts only once Android reports thermal status 0 (NONE), after a 30 s settle. The device is set to fixed performance mode, which is the official recommendation.

Performance results

Metric	Original / baseline	Arm-optimized	Improvement
Decode throughput (tok/s)	7.87	22.09	2.81 x
Time to first token (ms)	6346.48	983.29	6.45 x
End-to-end latency p50 (ms)	22608.92	6782.29	3.33 x
Peak memory (MB)	3712.12	2523.97	1.47 x
Model size (MB)	3600.80	1215.36	2.96 x

Accuracy

Accuracy was evaluated using the same preprocessing, input resolution, and evaluation protocol described below. Where possible, the optimized model is compared against the original model under the same evaluation conditions.

Evaluation setup

Field	Value
Dataset	edinburgh-dawg/mmlu-redux-2.0
Split	test, 0-shot
Number of samples	5330
Metric(s)	MMLU accuracy (%)
Evaluation runtime	llama.cpp

Accuracy results

Metric	Original / baseline	Arm-optimized	Change
MMLU accuracy (%)	63.9024	62.4578	-1.44 pp

Accuracy was measured using the evaluation setup described above. Users should re-evaluate the model on their own data before production use.

Arm optimization approach

Arm optimized this model for efficient inference on Arm-based platforms using a hardware-aware conversion and validation flow.

For this release, Arm used:

Optimization area	Applied?	Notes
Model conversion	Yes	Hugging Face checkpoint converted to GGUF; conversion passes --no-mtp to drop a multi-token-prediction draft head that tooling miscounts, leaving all 24 real transformer blocks intact
Quantization	Yes	Q4KM k-quant, imatrix-weighted, applied uniformly to the whole graph via llama-quantize; importance matrix collected with llama-imatrix over 200 wikitext samples; no layers excluded
Runtime/backend selection	Yes	llama.cpp with the ggml CPU backend, built with -march=armv8.6-a+dotprod+i8mm and GGMLCPUKLEIDIAI=ON
Graph/runtime compatibility updates	Yes	162 tensors kept at Q4K, 25 promoted to Q6K, and 133 norm / SSM conv and time-step tensors left in fp32
Accuracy validation	Yes	Compared against the original model or published baseline
Performance validation	Yes	Measured on the reference Arm platform

The goal of this process is to improve deployment characteristics such as latency, memory use, model size, and runtime compatibility while preserving the model's intended behavior. Detailed conversion scripts, calibration configuration, or backend-specific implementation details may be provided separately where appropriate.

Using this model

A representative evaluation target (vivo X300, Android) cannot run Python or uv, so example.py is validated on a realistic Arm Linux development host instead: aarch64 Ubuntu 24.04 (glibc 2.39), Python 3.12.

llama-cpp-python ships only as a source distribution, so uv sync compiles ggml on the host. That needs a C/C++ toolchain and CMake, declared in pyproject.toml and installed with the system package manager:

bash

sudo apt-get install -y build-essential cmake

pyproject.toml builds it with -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON -DGGML_BLAS=OFF -DGGML_OPENMP=ON, so the compiler selects whatever Arm features the build host has rather than the -march=armv8.6-a+dotprod+i8mm of the benchmarked Android build. The run is therefore a functional check of the inference flow, not a reproduction of the vivo X300 benchmark numbers above.

Note: The Python/uv example runs on AWS Graviton (Ubuntu arm64) to confirm runtime compatibility only, and is intended as a guideline for building an equivalent run script on Mobile CPU systems.

Install dependencies

Dependencies are declared in pyproject.toml and pinned in uv.lock, both of which ship with this repository. Resolve and install them into a local virtual environment with uv:

bash

uv python install
uv sync --frozen

Run the example

bash

uv run example.py

The example writes sample_input.txt and predictions.json next to itself. Both are committed here, so run it from a copy of this directory if you want to keep the known-good output intact.

Expected input

Property	Value
Input shape	[1, T], where T is the runtime sequence length
Input type	int32 token ids (llama.cpp llama_token)
Input range	Vocabulary ids in [0, 248320)
Preprocessing	Apply the chat template embedded in the `.gguf`, tokenize with the embedded BPE / gpt2 tokenizer and qwen35 pre-tokenizer, then truncate to the n_ctx of the runtime handle (2048 in the shipped recipe)

Expected output

Property	Value
Output shape	[1, T, 248320]
Output type	Per-token logits over the vocabulary when logits_all is requested; last-position logits only by default
Postprocessing	Decode with the tokenizer embedded in the `.gguf`; greedy sampling (temperature 0) in the shipped example, with the KV cache internal to llama.cpp and the sampled token fed back one at a time; stop on the eos token (id 248046) or max_tokens

Intended use

This model is intended for developers evaluating text generation workloads on Arm-based platforms. It is suitable as a reference implementation for benchmarking, prototyping, and integration exploration.

Limitations

—Performance depends on the target device, runtime version, backend/delegate support, memory configuration, and system load.
—Accuracy was evaluated on edinburgh-dawg/mmlu-redux-2.0 test and may not generalize to all domains.
—This release preserves the original model's intended task and behavior, but users should validate it for their own application, data, and deployment environment.
—This repository is not a replacement for the original model documentation.
—One device, one thread configuration. All performance figures come from a single vivo X300 at 4 threads pinned to CPU4-7 with taskset F0, the big-core cluster. Other Arm SoCs, thread counts, or thermal states will differ.
—The Python and uv example establishes runtime compatibility on Arm Linux only. It reports no latency or memory figure, and the numbers in the tables above come from the vivo X300 benchmark, not from that host.

Additional notes

The f16 reference is deliberate rather than bf16: ggml has no optimized bf16 CPU kernel, so a bf16 baseline would be measured through a scalar fallback and would inflate every speedup ratio above.

The accuracy table reports the mcf protocol (single-token letter logit).

Parameter counts summed over the tensors actually present in each GGUF: language-model tower (this release) 1,881.83 M, vision projector 331.42 M, full checkpoint 2,213.24 M.

About this version

Original Model: Qwen/Qwen3.5-2B by Alibaba Cloud (Qwen team) - Repository

Optimization/conversion: Arm-Optimized version for execution on Arm-based platforms.

Converted/optimized by: Arm

License: The Original Model and the Optimized Model are subject to Apache-2.0.

This repository contains a converted or optimized version of the Original Model (the “Optimized Model”). The Original Model has been converted or optimized as described above for execution on Arm-based platforms.

No retraining or fine-tuning of the Original Model was performed as part of the conversion or optimization. The conversion or optimization was not intended to change the Original Model’s behavior or intended use.

Original Model and Documentation

For information about the Original Model, including its development, training data, intended uses, limitations and other relevant information, please refer to the Original Model repository. Information in that repository was provided by the original developer or other third parties and, unless expressly stated otherwise, has not been independently verified by Arm.

Licenses and Third-Party Terms

Use of the Original Model and the Optimized Model is subject to the applicable licenses, usage restrictions and other terms identified above and in the relevant repositories. Publication of the Optimized Model does not grant any rights beyond those provided under the applicable license terms.

You are responsible for reviewing those terms and ensuring that your use of the Original Model and the Optimized Model is permitted.

Purpose of this Release

The Optimized Model is provided as a reference implementation to demonstrate and evaluate execution and performance on Arm-based systems. It is not a production-ready or supported solution.

Arm’s publication of the Optimized Model does not constitute an endorsement or certification of the Original Model or a representation that the Optimized Model is suitable for production use or any particular purpose.

To the fullest extent permitted by applicable law (i) the Optimized Model is provided “as is.” Arm makes no representations or warranties that the Original Model, the Optimized Model or their outputs are accurate, safe, secure, non-infringing, legally compliant, suitable for production use or fit for any particular purpose; and (ii) Arm will not be liable for any loss or damage arising from or in connection with the Optimized Model, its use or its outputs.

You are responsible for independently evaluating the Optimized Model, its outputs and its suitability for your intended use, including compliance with applicable legal, regulatory, safety and security requirements.

Arm does not commit to provide ongoing support, maintenance or updates for the Optimized Model. Any use of or reliance on the Optimized Model or its outputs is at your own risk.