Modelpublic

HY-2012/Qwen3-1.7B-GPTQ-Int4-NPU2

sourceHugging Faceapache-2.0updated 7d agoView on Hugging Face

0likes29downloads

Model Card

Qwen3-1.7B-GPTQ-Int4-NPU2

This version of Qwen3-1.7B-GPTQ-Int4 has been converted to run on the Axera NPU using w4a16 quantization, compiled with NPU2 (dual NPU-core) scheduling for AX650.

Compatible with Pulsar2 version: 7.0 (patch1)

Convert tools links:

For those who are interested in model conversion, you can try to export axmodel through the original repo : https://huggingface.co/Qwen/Qwen3-1.7B

The GPTQ-Int4 weights used for this build: https://huggingface.co/JunHowie/Qwen3-1.7B-GPTQ-Int4

Pulsar2 Link, How to Convert LLM from Huggingface to axmodel

AXera NPU LLM Runtime

Reference build (NPU default): https://huggingface.co/AXERA-TECH/Qwen3-1.7B-GPTQ-Int4

Convert the Huggingface Qwen3-1.7B-GPTQ-Int4 (AutoGPTQ layout, groupsize=128, sym) to axmodel with w4a16 quantization. Compared with the reference build, the only change is `--npumode NPU2`, i.e. the model is scheduled on 2 of the 3 NPU cores of AX650.

shell

export FLOAT_MATMUL_USE_CONV_EU=1 # only support AX650, for better performance, please set this env var before running the conversion command.

# context window size 2048, prefill length 1152 (step 128), NPU2 dual-core scheduling
pulsar2 llm_build2 --input_path Qwen3-1.7B-GPTQ-Int4 --output_path <your path> \
--hidden_state_type bf16 --weight_type s4 \
--max_context 2048 --prefill_len 1152 --prefill_step_size 128 \
--chip AX650 --npu_mode NPU2 -c 1 --parallel 32

Support Platform

Chips	NPU mode	w4a16	CMM	Flash
AX650	NPU2 (2 cores)	8.0 ~ 10.7 tokens/sec	~1.6 GiB	1.9 GiB

Measured on AX650N with axllm, greedy decode (default sampling disabled), short prompts, context 2048. TTFT is about 0.38 ~ 0.74 s. With NPU2 the model occupies 2 NPU cores, leaving the third core free for other tasks.

How to use

安装 axllm

方式一：克隆仓库后执行安装脚本：

shell

git clone -b axllm https://github.com/AXERA-TECH/ax-llm.git
cd ax-llm
./install.sh

方式二：一行命令安装（默认分支 axllm）：

shell

curl -fsSL https://raw.githubusercontent.com/AXERA-TECH/ax-llm/axllm/install.sh | bash

方式三：下载Github Actions CI 导出的可执行程序（适合没有编译环境的用户）：

如果没有编译环境，请到： https://github.com/AXERA-TECH/ax-llm/actions?query=branch%3Aaxllm 下载 最新 CI 导出的可执行程序（axllm），然后：

shell

chmod +x axllm
sudo mv axllm /usr/bin/axllm

模型下载（Hugging Face）

shell

mkdir -p HY-2012/Qwen3-1.7B-GPTQ-Int4-NPU2
cd HY-2012/Qwen3-1.7B-GPTQ-Int4-NPU2
hf download HY-2012/Qwen3-1.7B-GPTQ-Int4-NPU2 --local-dir .

# structure of the downloaded files
tree -L 2
.
├── README.md
├── config.json
├── model.embed_tokens.weight.bfloat16.bin
├── post_config.json
├── qwen3_p128_l0_together.axmodel
...
├── qwen3_p128_l27_together.axmodel
├── qwen3_post.axmodel
└── qwen3_tokenizer.txt

0 directories, 34 files

Inference with AX650 Host, such as M4N-Dock(爱芯派Pro) or AX650N DEMO Board

运行（CLI）

shell

(base) root@ax650:~# axllm run HY-2012/Qwen3-1.7B-GPTQ-Int4-NPU2/
INF Init:468 | LLM init start
tokenizer_type = 1
 96% | ##############################   |  30 /  31 [3.85s<3.98s, 7.79 count/s] init post axmodel ok,remain_cmm(8349 MB)
INF Init:791 | max_token_len : 2047
INF Init:794 | kv_cache_size : 1024, kv_cache_num: 2047
INF init_groups_from_model:150 | prefill_token_num : 128
INF init_groups_from_model:398 | decode grp: 0, gid: 0, max_token_len : 2047
INF init_groups_from_model:402 | prefill grp: 0, gid: 1, history_cap: 0, total_cap: 128, symbolic_cap: 1
INF init_groups_from_model:402 | prefill grp: 1, gid: 2, history_cap: 128, total_cap: 256, symbolic_cap: 128
INF init_groups_from_model:402 | prefill grp: 2, gid: 3, history_cap: 256, total_cap: 384, symbolic_cap: 256
INF init_groups_from_model:402 | prefill grp: 3, gid: 4, history_cap: 384, total_cap: 512, symbolic_cap: 384
INF init_groups_from_model:402 | prefill grp: 4, gid: 5, history_cap: 512, total_cap: 640, symbolic_cap: 512
INF init_groups_from_model:402 | prefill grp: 5, gid: 6, history_cap: 640, total_cap: 768, symbolic_cap: 640
INF init_groups_from_model:402 | prefill grp: 6, gid: 7, history_cap: 768, total_cap: 896, symbolic_cap: 768
INF init_groups_from_model:402 | prefill grp: 7, gid: 8, history_cap: 896, total_cap: 1024, symbolic_cap: 896
INF init_groups_from_model:402 | prefill grp: 8, gid: 9, history_cap: 1024, total_cap: 1152, symbolic_cap: 1024
INF init_groups_from_model:409 | prefill_max_token_num : 1152
INF Init:27 | LLaMaEmbedSelector use mmap
100% | ################################ |  31 /  31 [3.85s<3.85s, 8.04 count/s] embed_selector init ok
INF Init:1139 | LLM init ok
Commands:
  /q, /exit  退出
  /reset     重置 kvcache
  /dd        删除一轮对话
  /pp        打印历史对话
Ctrl+C: 停止当前生成
----------------------------------------
prompt >> 用一句话介绍一下杭州
INF SetKVCache:2064 | decode_grpid:0 prefill_grpid:1 history_cap:0 total_cap:128 symbolic_cap:1 precompute_len:0 input_num_token:16 prefer_symbolic_group:0
INF SetKVCache:2086 | current prefill_max_token_num:1152
INF Run:257 | input token num : 16, prefill_split_num : 1
INF Run:658 | ttft: 442.56 ms (prepare=249.00 ms, text=193.55 ms)
杭州是中国浙江省会，以西湖美景和历史文化闻名，是江南四大名镇之一。
NTC Run:1090 | decode stop: reason=eos prompt_tokens=16 generated_tokens=19 output_max_token=150 max_token_len=2047 context_remaining=2012 avg 8.28 token/s
prompt >> /q

启动服务（OpenAI 兼容）

shell

(base) root@ax650:~# axllm serve HY-2012/Qwen3-1.7B-GPTQ-Int4-NPU2/ --port 8080
INF Init:468 | LLM init start
... init 31/31 axmodel ok, remain_cmm(8349 MB)
INF Init:791 | max_token_len : 2047
INF Init:794 | kv_cache_size : 1024, kv_cache_num: 2047
INF init_groups_from_model:409 | prefill_max_token_num : 1152
Starting server on port 8080 with model 'Qwen3-1.7B-AX650-NPU2'...
OpenAI API Server starting on http://0.0.0.0:8080
Max concurrency: 1
Models: Qwen3-1.7B-AX650-NPU2

服务启动完成后，即可通过标准 OpenAI 兼容接口进行调用。

API 调用示例

服务启动完成后，可直接使用标准 HTTP 请求访问 OpenAI 兼容接口。最简调用命令如下：

shell

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen3-1.7B-AX650-NPU2","messages":[{"role":"user","content":"你好"}]}'

如需使用 ax-llm 项目中的示例脚本进行测试，可参考如下方式：

text

root@ax650:~/llm-test# curl -sOL https://raw.githubusercontent.com/AXERA-TECH/ax-llm/refs/heads/axllm/scripts/openai_demo.py

root@ax650:~/llm-test# python openai_demo.py --model Qwen3-1.7B-AX650-NPU2 --api_url http://127.0.0.1:8080/v1

如需关闭思考模式（Qwen3 thinking），可在请求中追加 "chat_template_kwargs":{"enable_thinking":false}。

需要说明的是，openai_demo.py 仅作为接口调用示例，实际应用中推荐直接按照 OpenAI 接口规范进行调用对接。

板端运行程序编译流程，以及更多 run / serve / API 使用细节，请参考我们在 Github 上的开源项目 AX-LLM