CoolFace
Modelpublic

AXERA-TECH/Qwen3.5-2B-AX650-GPTQ-Int4-C256-P6K-CTX8K

sourceHugging Facemitupdated 27d agoView on Hugging Face
1likes45downloads
Model Card

Qwen3.5-2B

This version of Qwen3.5-2B has been converted to run on the Axera NPU using w4a16 quantization.

Compatible with Pulsar2 version: 5.0

Convert tools links:

For those who are interested in model conversion, you can try to export axmodel through the original repo :

  • —https://huggingface.co/Qwen/Qwen3.5-2B

Pulsar2 Link, How to Convert LLM from Huggingface to axmodel

AXera NPU HOST LLM Runtime

Support Platform

Image Process |Chips| input size | image num | ttft(168 tokens) | w4a16 | CMM | Flash | |--|--|--|--|--|--|--| |AX650| 384*384 | 1 | 368 ms | 13.4 tokens/sec| 1.90GiB | 2.86GiB |

Video Process |Chips| input size | image num | ttft(600 tokens) | w4a16 | CMM | Flash | |--|--|--|--|--|--|--| |AX650| 384*384 | 8 | 892 ms | 13.4 tokens/sec| 1.90GiB | 2.86GiB |

The DDR capacity refers to the CMM memory that needs to be consumed. Ensure that the CMM memory allocation on the development board is greater than this value.

How to use

安装 axllm

方式一：克隆仓库后执行安装脚本：

shell
git clone -b axllm https://github.com/AXERA-TECH/ax-llm.git
cd ax-llm
./install.sh

方式二：一行命令安装（默认分支 axllm）：

shell
curl -fsSL https://raw.githubusercontent.com/AXERA-TECH/ax-llm/axllm/install.sh | bash

方式三：下载Github Actions CI 导出的可执行程序（适合没有编译环境的用户）：

如果没有编译环境，请到： https://github.com/AXERA-TECH/ax-llm/actions?query=branch%3Aaxllm 下载 最新 CI 导出的可执行程序（axllm），然后：

shell
chmod +x axllm
sudo mv axllm /usr/bin/axllm

模型下载（Hugging Face）

先创建模型目录并进入，然后下载到该目录：

shell
mkdir -p AXERA-TECH/Qwen3.5-2B-AX650-GPTQ-Int4-C256-P6K-CTX8K
cd AXERA-TECH/Qwen3.5-2B-AX650-GPTQ-Int4-C256-P6K-CTX8K
hf download AXERA-TECH/Qwen3.5-2B-AX650-GPTQ-Int4-C256-P6K-CTX8K --local-dir .

# structure of the downloaded files
tree -L 3
`-- AXERA-TECH
    `-- AXERA-TECH/Qwen3.5-2B-AX650-GPTQ-Int4-C256-P6K-CTX8K
        |-- qwen3_5_vision.axmodel
        |-- README.md
        |-- config.json
        |-- image.png
        |-- model.embed_tokens.weight.bfloat16.bin
        |-- post_config.json
        |-- qwen3_5_tokenizer.txt
        |-- qwen3_5_text_p256_l0_together.axmodel
        ...
        |-- qwen3_5_text_p256_l23_together.axmodel
        |-- qwen3_5_text_post.axmodel
        `-- vision_cache

3 directories, 39 files

Inference with AX650 Host, such as M4N-Dock(爱芯派Pro) or AX650N DEMO Board

运行（CLI）

图片理解
shell
root@ax650 ~/yongqiang/lhj # axllm run Qwen3.5-2B-AX650-GPTQ-Int4-C256-P6K-CTX8K/
10:44:41.266 INF Init:2700 | LLM init start
10:44:41.272 INF Init:2720 | mixed attention enabled: full_attention_interval=4 ref_full_layer_idx=3
tokenizer_type = 3
huggingface tokenizer mode = gpt2_byte_bpe
 96% | ##############################   |  26 /  27 [25.48s<26.47s, 1.02 count/s] init post axmodel ok,remain_cmm(8076 MB)
10:45:06.757 INF Init:3023 | max_token_len : 2047
10:45:06.757 INF Init:3026 | kv_cache_size : 512, kv_cache_num: 2047
10:45:06.757 INF init_groups_from_model:1827 | prefill_token_num : 256
10:45:06.757 INF init_groups_from_model:2075 | decode grp: 0, gid: 0, max_token_len : 2047
10:45:06.757 INF init_groups_from_model:2075 | decode grp: 1, gid: 1, max_token_len : 4095
10:45:06.757 INF init_groups_from_model:2075 | decode grp: 2, gid: 2, max_token_len : 6143
10:45:06.757 INF init_groups_from_model:2075 | decode grp: 3, gid: 3, max_token_len : 8191
10:45:06.757 INF init_groups_from_model:2079 | prefill grp: 0, gid: 4, history_cap: 0, total_cap: 256, symbolic_cap: 1
10:45:06.757 INF init_groups_from_model:2079 | prefill grp: 1, gid: 5, history_cap: 256, total_cap: 512, symbolic_cap: 256
...
...
10:45:06.758 INF init_groups_from_model:2079 | prefill grp: 24, gid: 28, history_cap: 6144, total_cap: 6400, symbolic_cap: 6144
10:45:06.758 INF init_groups_from_model:2086 | prefill_max_token_num : 6400
10:45:06.758 INF init_layer_groups:1293 | layer 0 decode groups=1 ref=4, reuse gid 0 for later decode shapes
10:45:06.758 INF init_layer_groups:1306 | layer 0 prefill groups=2 ref=25, reuse gid 2 for later prefill chunks
...
...
10:45:06.759 INF init_layer_groups:1293 | layer 22 decode groups=1 ref=4, reuse gid 0 for later decode shapes
10:45:06.759 INF init_layer_groups:1306 | layer 22 prefill groups=2 ref=25, reuse gid 2 for later prefill chunks
10:45:06.759 INF Init:27 | LLaMaEmbedSelector use mmap
100% | ################################ |  27 /  27 [25.49s<25.49s, 1.06 count/s] embed_selector init ok
10:45:12.042 INF Init:1647 | Qwen token ids: vision_start=248053 image_pad=248056 video_pad=248057 audio_pad=-1
10:45:12.042 INF Init:1704 | VisionModule init ok: type=Qwen3VL, tokens_per_block=144, embed_size=2048, out_dtype=fp32
10:45:12.042 WRN Init:1713 | Vision preprocess backend: SimpleCV (OpenCV not found at build time; minor differences vs OpenCV are possible)
10:45:12.042 INF load_config:554 | load config:
10:45:12.042 INF load_config:554 | {
10:45:12.042 INF load_config:554 |     "enable_repetition_penalty": false,
10:45:12.042 INF load_config:554 |     "enable_temperature": false,
10:45:12.042 INF load_config:554 |     "enable_top_k_sampling": false,
10:45:12.042 INF load_config:554 |     "enable_top_p_sampling": false,
10:45:12.042 INF load_config:554 |     "penalty_window": 20,
10:45:12.042 INF load_config:554 |     "repetition_penalty": 1.2,
10:45:12.042 INF load_config:554 |     "temperature": 0.9,
10:45:12.042 INF load_config:554 |     "top_k": 10,
10:45:12.042 INF load_config:554 |     "top_p": 0.8
10:45:12.042 INF load_config:554 | }
10:45:12.043 INF Init:3138 | LLM init ok
Commands:
  /q, /exit  退出
  /reset     重置 kvcache
  /dd        删除一轮对话
  /pp        打印历史对话
Ctrl+C: 停止当前生成
VLM enabled: after each prompt, input media path (empty = text-only). Use "video:<frames_dir>" or "video:<video_file>[:<fps>]" for video (default fps=2), "audio:<file>" for audio.
----------------------------------------
prompt >> describe the image
image >> image.png
10:49:05.178 INF EncodeForContent:2344 | Qwen-VL pixel_values[0] bytes=884736 min=0 max=238 (w=384 h=384 tp=2 ps=16 sm=2)
10:49:05.328 INF EncodeForContent:2403 | vision cache store: /mnt/sdcard/sunyijie/projects/Qwen3.5-2B-AX650-GPTQ-Int4-C256-P6K-CTX8K/assets/image.png
10:49:05.333 INF SetKVCache:3721 | decode_grpid:0 prefill_grpid:4 history_cap:0 total_cap:256 symbolic_cap:1 precompute_len:0 input_num_token:168 prefer_symbolic_group:0
10:49:05.333 INF SetKVCache:3743 | current prefill_max_token_num:6400
10:49:05.430 INF SetKVCache:3759 | first run
10:49:05.455 INF Run:4287 | input token num : 168, prefill_split_num : 1
10:49:05.455 INF Run:4367 | prefill chunk p=0 history_len=0 grpid=4 kv_cache_num=0 input_tokens=168
10:49:05.898 INF Run:4680 | ttft: 443.24 ms
<think>10:49:05.898 INF Run:4722 | VLM decode positions: rope_start=36 dense_kv_start=168

The user wants a description of the provided image.

1.  **Identify the main subjects:** There are three figures in the image. They are wearing white, bulky suits that look like astronaut space suits.
2.  **Analyze the setting:** They are standing in a forest. There are tall, thin trees (looks like bamboo or similar tall grasses) in the background. The ground is covered in low-lying vegetation, possibly grass or weeds.
3.  **Analyze the poses and actions:**
    *   **Center figure:** This figure is standing upright, facing forward. Their arms are raised, with hands up near their chest/shoulders. They seem to be looking towards the viewer or slightly to the side. Their helmet visor is reflective.
    *   **Left figure:** This figure is standing slightly behind and to the left of the center figure. They are facing forward, looking slightly to the right. Their hands are near their waist/hips.
    *   **Right figure:** This figure is standing to the right, slightly bent forward or leaning. They are looking towards the center figure or slightly past them. Their posture suggests movement or perhaps looking at something in front.
4.  **Analyze the lighting and color:** The image has a cool, desaturated color palette. It looks like a monochromatic or sepia tone, or perhaps a very desaturated color grading. The lighting is somewhat soft, with shadows in the background. The helmets reflect light, suggesting a shiny visor.
5.  **Analyze the style:** The image has a slightly surreal or AI-generated quality. The suits look a bit too perfect or "clean" compared to real astronaut suits, which often have more scuffs or specific markings. The trees in the background look a bit like bamboo stalks. The overall aesthetic is somewhat ethereal or dreamlike.

**Drafting the description:**
*   Start with the main subject: Three astronauts in white space suits.
*   Describe their positions: Center one standing with arms up, left one standing, right one bent/leaning.
*   Describe the environment: Forest, tall thin trees (bamboo-like), grassy ground.
*   Describe the atmosphere: Cool tones, desaturated, reflective visors.
*   Mention the potential AI nature: The suits look a bit idealized.

**Refining the description:**
The image features three figures dressed in white, bulky astronaut space suits standing in a forest setting. The central figure is standing upright with both arms raised, looking forward with a reflective visor. To the left, another figure stands facing forward, and to the right, a third figure is bent slightly forward, looking towards the center. The background consists of tall, thin trees resembling bamboo stalks, and the ground is covered in low-lying vegetation. The overall color palette is desaturated and cool, giving the scene a somewhat ethereal or monochromatic appearance. The visors on the helmets reflect light, adding a shiny element to the suits.

Let's double check the "AI" aspect. The prompt says "you are a helpful assistant." I should just describe what is there. The description should be objective.

Final plan:
1.  Describe the three astronauts.
2.  Describe their poses.
3.  Describe the environment (forest, trees, ground).
4.  Describe the color/style (desaturated, cool tones).
5.  Mention the reflective visors.

This covers all key points.
</think>

This image depicts a surreal scene featuring three figures dressed in white, bulky astronaut space suits standing in a forest.

*   **The Figures:**
    *   **Center:** The central figure stands upright, facing forward with both arms raised. Their helmet visor is reflective, catching the light.
    *   **Left:** To the left of the center, a second figure stands facing forward, looking slightly to the right.
    *   **Right:** To the right, a third figure is bent slightly forward, looking towards the center figure.
*   **The Setting:** They are standing in a forest with tall, thin trees that resemble bamboo stalks. The ground is covered in low-lying vegetation, possibly grass or weeds.
*   **Atmosphere:** The image has a desaturated, cool color palette, giving it a somewhat monochromatic or sepia tone. The lighting is soft, and the visors on the helmets reflect light, adding a shiny element to the suits. The overall aesthetic has a slightly surreal or dreamlike quality.

10:50:17.673 NTC Run:5095 | hit eos,decode avg 13.25 token/s
10:50:17.673 INF GetKVCache:3698 | precompute_len:1120, remaining:5280 (tracked)
视频理解
shell
root@ax650 ~/Qwen3.5-2B-AX650-GPTQ-Int4-C256-P6K-CTX8K # ./axllm run ./
10:44:41.266 INF Init:2700 | LLM init start
10:44:41.272 INF Init:2720 | mixed attention enabled: full_attention_interval=4 ref_full_layer_idx=3
tokenizer_type = 3
huggingface tokenizer mode = gpt2_byte_bpe
 96% | ##############################   |  26 /  27 [25.48s<26.47s, 1.02 count/s] init post axmodel ok,remain_cmm(8076 MB)
10:45:06.757 INF Init:3023 | max_token_len : 2047
10:45:06.757 INF Init:3026 | kv_cache_size : 512, kv_cache_num: 2047
10:45:06.757 INF init_groups_from_model:1827 | prefill_token_num : 256
10:45:06.757 INF init_groups_from_model:2075 | decode grp: 0, gid: 0, max_token_len : 2047
10:45:06.757 INF init_groups_from_model:2075 | decode grp: 1, gid: 1, max_token_len : 4095
10:45:06.757 INF init_groups_from_model:2075 | decode grp: 2, gid: 2, max_token_len : 6143
10:45:06.757 INF init_groups_from_model:2075 | decode grp: 3, gid: 3, max_token_len : 8191
10:45:06.757 INF init_groups_from_model:2079 | prefill grp: 0, gid: 4, history_cap: 0, total_cap: 256, symbolic_cap: 1
10:45:06.757 INF init_groups_from_model:2079 | prefill grp: 1, gid: 5, history_cap: 256, total_cap: 512, symbolic_cap: 256
...
...
10:45:06.758 INF init_layer_groups:1293 | layer 2 decode groups=1 ref=4, reuse gid 0 for later decode shapes
10:45:06.758 INF init_layer_groups:1306 | layer 2 prefill groups=2 ref=25, reuse gid 2 for later prefill chunks
...
...
10:45:06.759 INF init_layer_groups:1293 | layer 22 decode groups=1 ref=4, reuse gid 0 for later decode shapes
10:45:06.759 INF init_layer_groups:1306 | layer 22 prefill groups=2 ref=25, reuse gid 2 for later prefill chunks
10:45:06.759 INF Init:27 | LLaMaEmbedSelector use mmap
100% | ################################ |  27 /  27 [25.49s<25.49s, 1.06 count/s] embed_selector init ok
10:45:12.042 INF Init:1647 | Qwen token ids: vision_start=248053 image_pad=248056 video_pad=248057 audio_pad=-1
10:45:12.042 INF Init:1704 | VisionModule init ok: type=Qwen3VL, tokens_per_block=144, embed_size=2048, out_dtype=fp32
10:45:12.042 WRN Init:1713 | Vision preprocess backend: SimpleCV (OpenCV not found at build time; minor differences vs OpenCV are possible)
10:45:12.042 INF load_config:554 | load config:
10:45:12.042 INF load_config:554 | {
10:45:12.042 INF load_config:554 |     "enable_repetition_penalty": false,
10:45:12.042 INF load_config:554 |     "enable_temperature": false,
10:45:12.042 INF load_config:554 |     "enable_top_k_sampling": false,
10:45:12.042 INF load_config:554 |     "enable_top_p_sampling": false,
10:45:12.042 INF load_config:554 |     "penalty_window": 20,
10:45:12.042 INF load_config:554 |     "repetition_penalty": 1.2,
10:45:12.042 INF load_config:554 |     "temperature": 0.9,
10:45:12.042 INF load_config:554 |     "top_k": 10,
10:45:12.042 INF load_config:554 |     "top_p": 0.8
10:45:12.042 INF load_config:554 | }
10:45:12.043 INF Init:3138 | LLM init ok
Commands:
  /q, /exit  退出
  /reset     重置 kvcache
  /dd        删除一轮对话
  /pp        打印历史对话
Ctrl+C: 停止当前生成
VLM enabled: after each prompt, input media path (empty = text-only). Use "video:<frames_dir>" or "video:<video_file>[:<fps>]" for video (default fps=2), "audio:<file>" for audio.
----------------------------------------
prompt >> 描述视频内容
media >> video:assets/football.mp4:1
11:02:29.955 INF extract_video_frames_ffmpeg:405 | Extracting raw video container to frames: /mnt/sdcard/sunyijie/projects/Qwen3.5-2B-AX650-GPTQ-Int4-C256-P6K-CTX8K/assets/football.mp4 -> /tmp/axllm_video_frames/video_321358652441300_0 fps=0.99900099900099892
11:02:38.888 INF collect_video_frame_paths:485 | Video fps sampling: path=/mnt/sdcard/sunyijie/projects/Qwen3.5-2B-AX650-GPTQ-Int4-C256-P6K-CTX8K/assets/football.mp4 fps=1 duration=60.060s target_frames=60 selected=60
11:02:52.817 INF SetKVCache:3721 | decode_grpid:2 prefill_grpid:4 history_cap:0 total_cap:256 symbolic_cap:1 precompute_len:0 input_num_token:4344 prefer_symbolic_group:0
11:02:52.818 INF SetKVCache:3743 | current prefill_max_token_num:6400
11:02:52.916 INF SetKVCache:3759 | first run
11:02:52.931 INF Run:4287 | input token num : 4344, prefill_split_num : 17
11:02:52.931 INF Run:4367 | prefill chunk p=0 history_len=0 grpid=4 kv_cache_num=0 input_tokens=256
11:02:53.349 INF Run:4367 | prefill chunk p=1 history_len=256 grpid=6 kv_cache_num=512 input_tokens=256
...
...
11:03:00.332 INF Run:4367 | prefill chunk p=15 history_len=3840 grpid=20 kv_cache_num=4096 input_tokens=256
11:03:00.874 INF Run:4367 | prefill chunk p=16 history_len=4096 grpid=21 kv_cache_num=4352 input_tokens=248
11:03:01.476 INF Run:4680 | ttft: 8545.62 ms
<think>11:03:01.476 INF Run:4722 | VLM decode positions: rope_start=4212 dense_kv_start=4344


</think>

好的，这张视频截图展示了一个足球比赛的精彩瞬间，并配有解说员的反应。以下是详细描述：

- **主要画面**：
    - 视频的主体部分是一个足球比赛的实时画面。
    - 比赛场地是绿色的草坪，画有标准的白色界线。
    - 场上有两名球队：一支穿着白色球衣的球队和一支穿着紫红色条纹球衣的球队。
    - 画面捕捉到了一个进球后的瞬间。一名身穿紫红色球衣的球员（背号9号，根据上下文应为哈维·阿隆索）在球门附近庆祝，他张开双臂，表情激动。
    - 在他身后，他的队友们正在拥抱庆祝。
    - 在画面下方，可以看到一个足球被踢入球门，守门员在球门内倒地。

- **次要画面**：
    - 在右上角，有一个小窗口，显示着一位男性解说员。
    - 他留着黑色短发，戴着黑色无檐帽，身穿黄黑相间的条纹球衣。
    - 他正对着镜头说话，并用手势辅助表达，看起来非常兴奋和投入。
    - 他的背景墙上挂着一些奖杯和照片。

- **视频信息**：
    - 视频左上角的文字显示：“Barcelona vs Sevilla 6-1 All Goals & Extended Highlights 30/01/2019 HD”。这表明这是巴塞罗那队以6-1比分战胜塞维利亚队的比赛，包含了所有进球和延长时间的精彩集锦，录制于2019年1月30日。
    - 视频左下角有“FC2019MPS10”的水印，表明这是FIFA 2019游戏（或类似足球游戏）的截图。

11:03:32.662 NTC Run:5095 | hit eos,decode avg 11.93 token/s

启动服务（OpenAI 兼容）

shell
root@ax650:~# axllm serve AXERA-TECH/Qwen3.5-2B-AX650-GPTQ-Int4-C256-P6K-CTX8K
11:05:55.923 INF run_server_mode:2135 | server request/queue timeout: 300000 ms (config.server_timeout_ms, overridable via --server_timeout_ms)
11:05:55.925 INF Init:2700 | LLM init start
11:05:55.931 INF Init:2720 | mixed attention enabled: full_attention_interval=4 ref_full_layer_idx=3
tokenizer_type = 3
huggingface tokenizer mode = gpt2_byte_bpe
 96% | ##############################   |  26 /  27 [3.10s<3.22s, 8.37 count/s] init post axmodel ok,remain_cmm(8076 MB)
11:05:59.036 INF Init:3023 | max_token_len : 2047
11:05:59.036 INF Init:3026 | kv_cache_size : 512, kv_cache_num: 2047
11:05:59.036 INF init_groups_from_model:1827 | prefill_token_num : 256
11:05:59.036 INF init_groups_from_model:2075 | decode grp: 0, gid: 0, max_token_len : 2047
11:05:59.036 INF init_groups_from_model:2075 | decode grp: 1, gid: 1, max_token_len : 4095
11:05:59.036 INF init_groups_from_model:2075 | decode grp: 2, gid: 2, max_token_len : 6143
11:05:59.036 INF init_groups_from_model:2075 | decode grp: 3, gid: 3, max_token_len : 8191
11:05:59.036 INF init_groups_from_model:2079 | prefill grp: 0, gid: 4, history_cap: 0, total_cap: 256, symbolic_cap: 1
...
...
11:05:59.037 INF init_groups_from_model:2079 | prefill grp: 24, gid: 28, history_cap: 6144, total_cap: 6400, symbolic_cap: 6144
11:05:59.037 INF init_groups_from_model:2086 | prefill_max_token_num : 6400
11:05:59.037 INF init_layer_groups:1293 | layer 0 decode groups=1 ref=4, reuse gid 0 for later decode shapes
11:05:59.037 INF init_layer_groups:1306 | layer 0 prefill groups=2 ref=25, reuse gid 2 for later prefill chunks
...
...
11:05:59.038 INF init_layer_groups:1293 | layer 22 decode groups=1 ref=4, reuse gid 0 for later decode shapes
11:05:59.038 INF init_layer_groups:1306 | layer 22 prefill groups=2 ref=25, reuse gid 2 for later prefill chunks
11:05:59.038 INF Init:27 | LLaMaEmbedSelector use mmap
100% | ################################ |  27 /  27 [3.11s<3.11s, 8.69 count/s] embed_selector init ok
11:05:59.314 INF Init:1647 | Qwen token ids: vision_start=248053 image_pad=248056 video_pad=248057 audio_pad=-1
11:05:59.314 INF Init:1704 | VisionModule init ok: type=Qwen3VL, tokens_per_block=144, embed_size=2048, out_dtype=fp32
11:05:59.315 WRN Init:1713 | Vision preprocess backend: SimpleCV (OpenCV not found at build time; minor differences vs OpenCV are possible)
11:05:59.315 INF load_config:554 | load config:
11:05:59.315 INF load_config:554 | {
11:05:59.315 INF load_config:554 |     "enable_repetition_penalty": false,
11:05:59.315 INF load_config:554 |     "enable_temperature": false,
11:05:59.315 INF load_config:554 |     "enable_top_k_sampling": false,
11:05:59.315 INF load_config:554 |     "enable_top_p_sampling": false,
11:05:59.315 INF load_config:554 |     "penalty_window": 20,
11:05:59.315 INF load_config:554 |     "repetition_penalty": 1.2,
11:05:59.315 INF load_config:554 |     "temperature": 0.9,
11:05:59.315 INF load_config:554 |     "top_k": 10,
11:05:59.315 INF load_config:554 |     "top_p": 0.8
11:05:59.315 INF load_config:554 | }
11:05:59.315 INF Init:3138 | LLM init ok
Starting server on port 8000 with model 'AXERA-TECH/Qwen3.5-2B'...
API URLs:
  GET  http://127.0.0.1:8000/health
  GET  http://127.0.0.1:8000/v1/models
  POST http://127.0.0.1:8000/v1/chat/completions
  GET  http://10.126.102.188:8000/health
  GET  http://10.126.102.188:8000/v1/models
  POST http://10.126.102.188:8000/v1/chat/completions
  GET  http://10.126.103.31:8000/health
  GET  http://10.126.103.31:8000/v1/models
  POST http://10.126.103.31:8000/v1/chat/completions
  GET  http://172.17.0.1:8000/health
  GET  http://172.17.0.1:8000/v1/models
  POST http://172.17.0.1:8000/v1/chat/completions
Aliases:
  GET  http://127.0.0.1:8000/models
  POST http://127.0.0.1:8000/chat/completions
  GET  http://10.126.102.188:8000/models
  POST http://10.126.102.188:8000/chat/completions
  GET  http://10.126.103.31:8000/models
  POST http://10.126.103.31:8000/chat/completions
  GET  http://172.17.0.1:8000/models
  POST http://172.17.0.1:8000/chat/completions
OpenAI API Server starting on http://0.0.0.0:8000
Max concurrency: 1
Models: Qwen3.5-2B-AX650-GPTQ-Int4-C256-P6K-CTX8K

OpenAI 调用示例

python
from openai import OpenAI

API_URL = "http://127.0.0.1:8000/v1"
MODEL = "AXERA-TECH/Qwen3.5-2B-AX650-GPTQ-Int4-C256-P6K-CTX8K"

messages = [
    {"role": "system", "content": [{"type": "text", "text": "you are a helpful assistant."}]},
    {"role": "user", "content": "hello"},
]

client = OpenAI(api_key="not-needed", base_url=API_URL)
completion = client.chat.completions.create(
    model=MODEL,
    messages=messages,
)

print(completion.choices[0].message.content)

OpenAI 流式调用示例

python
from openai import OpenAI

API_URL = "http://127.0.0.1:8000/v1"
MODEL = "AXERA-TECH/Qwen3.5-2B-AX650-GPTQ-Int4-C256-P6K-CTX8K"

messages = [
    {"role": "system", "content": [{"type": "text", "text": "you are a helpful assistant."}]},
    {"role": "user", "content": "hello"},
]

client = OpenAI(api_key="not-needed", base_url=API_URL)
stream = client.chat.completions.create(
    model=MODEL,
    messages=messages,
    stream=True,
)

print("assistant:")
for ev in stream:
    delta = getattr(ev.choices[0], "delta", None)
    if delta and getattr(delta, "content", None):
        print(delta.content, end="", flush=True)
print("
")