CoolFace
Modelpublic

HY-2012/QwenPaw-Flash-2B-GPTQ-Int4-NPU2

sourceHugging Faceapache-2.0updated 7d agoView on Hugging Face
0likes39downloads
Model Card

QwenPaw-Flash-2B-GPTQ-Int4-NPU2

AX650 axmodel build of QwenPaw-Flash-2B (Qwen3.5-2B VLM, hybrid GatedDeltaNet + Gated Attention), compiled from the GPTQ-Int4 checkpoint HY-2012/QwenPaw-Flash-2B-GPTQ-int4 with w4a16 quantization.

This repository contains two NPU scheduling variants, both validated on AX650N:

VariantPath`npu_mode`Board status (AX650N, SDK V3.10.2)
NPU2 (2 cores)repo rootNPU2✅ text + image inference validated
NPU3 (3 cores)`NPU3/`NPU3✅ text + image inference validated
NPU2 leaves one NPU core free for other tasks (e.g. ASR/TTS pipelines) with almost no throughput loss.

Build details (both variants)

  • —Tool: Pulsar2 7.0 (patch1), image pulsar2:7.0-patch1
  • —Text: pulsar2 llm_build2 --weight_type s4 --hidden_state_type bf16 --max_context 2048 --prefill_len 1152 --prefill_step_size 128 --chip AX650 --npu_mode {NPU2|NPU3}
  • —Vision: exported vision tower ONNX (384×384, single output, no deepstack), pulsar2 build with U16 quantization
  • —Files: 24 × qwen3_5_text_p128_l*_together.axmodel, qwen3_5_text_post.axmodel, qwen3_5_vision.axmodel, model.embed_tokens.weight.bfloat16.bin, qwen3_5_tokenizer.txt (tokenizertype `Qwen35VL), config.json, post_config.json`

Measured on AX650N (SDK V3.10.2)

TestNPU2 (root)NPU3 (`NPU3/`)
Model load27/27 axmodels ok, ~5.3 min, CMM used ≈ 1.44 GiB27/27 ok, ~6.1 min, CMM used ≈ 1.45 GiB
Text (Chinese, 35 tokens)TTFT 922 ms, 8.46 tokens/sTTFT 859 ms, 8.78 tokens/s
Image (384×384, 150 tokens)TTFT 1995 ms (incl. vision), 9.87 tokens/sTTFT 1844 ms, 10.28 tokens/s
Vision moduletype=Qwen3VL, tokens_per_block=144, embed_size=2048, out_dtype=fp32same
Tip: if model loading fails right after a previously killed axllm process, reboot the board to reset NPU driver state, then load normally (seen once on a test board with stale state).

How to use

安装 axllm

方式一：克隆仓库后执行安装脚本：

shell
git clone -b axllm https://github.com/AXERA-TECH/ax-llm.git
cd ax-llm
./install.sh

方式二：一行命令安装（默认分支 axllm）：

shell
curl -fsSL https://raw.githubusercontent.com/AXERA-TECH/ax-llm/axllm/install.sh | bash

方式三：下载Github Actions CI 导出的可执行程序（适合没有编译环境的用户）：

如果没有编译环境，请到： https://github.com/AXERA-TECH/ax-llm/actions?query=branch%3Aaxllm 下载 最新 CI 导出的可执行程序（axllm），然后：

shell
chmod +x axllm
sudo mv axllm /usr/bin/axllm

模型下载（Hugging Face）

shell
mkdir -p HY-2012/QwenPaw-Flash-2B-GPTQ-Int4-NPU2
cd HY-2012/QwenPaw-Flash-2B-GPTQ-Int4-NPU2

# 默认下载根目录 = NPU2 版本
hf download HY-2012/QwenPaw-Flash-2B-GPTQ-Int4-NPU2 --local-dir .

# 或只下载 NPU3 版本
hf download HY-2012/QwenPaw-Flash-2B-GPTQ-Int4-NPU2 --include "NPU3/*" --local-dir .

Inference with AX650

运行（文本 + 图片，CLI）

shell
root@ax650:~# axllm run HY-2012/QwenPaw-Flash-2B-GPTQ-Int4-NPU2/
...
prompt >> 用一句话介绍一下杭州
杭州，这座位于中国东南沿海的江南城市，以西湖山水、灵隐寺古刹和苏杭文化闻名，是融合历史底蕴与现代活力的宜居之地。
prompt >> 描述一下这张图片。
image >> /path/to/image.jpg
这张图片捕捉了一个温馨、宁静的海滩场景，时间似乎是日落时分，光线柔和而温暖。...

启动服务（OpenAI 兼容）

shell
root@ax650:~# axllm serve HY-2012/QwenPaw-Flash-2B-GPTQ-Int4-NPU2/ --port 8080
...
VisionModule init ok: type=Qwen3VL, tokens_per_block=144, embed_size=2048, out_dtype=fp32
OpenAI API Server starting on http://0.0.0.0:8080
Models: QwenPaw-Flash-2B-AX650-NPU2

API 调用示例（文本）

shell
curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"QwenPaw-Flash-2B-AX650-NPU2","messages":[{"role":"user","content":"你好"}]}'

API 调用示例（图文）

shell
curl -sOL https://raw.githubusercontent.com/AXERA-TECH/ax-llm/refs/heads/axllm/scripts/openai_demo.py
python openai_demo.py --model QwenPaw-Flash-2B-AX650-NPU2 --api_url http://127.0.0.1:8080/v1 \
  --image /path/to/image.jpg --prompt "描述一下这张图片"

如需关闭思考模式（Qwen3.5 thinking），可在请求中追加 "chat_template_kwargs":{"enable_thinking":false}。

Related