Modelpublic

WaveMatrix/PaddleOCR-VL-1.5

sourceHugging Facebsd-3-clauseupdated 20d agoView on Hugging Face

0likes20downloads

Model Card

PaddleOCR-VL-1.5

This version of PaddleOCR-VL-1.5 has been converted to run on the WaveMatrix NPU using w4a16 quantization.

Compatible with Pulsar2 version: 5.0

Convert tools links:

For those who are interested in model conversion, you can try to export axmodel through the original repo:

—https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.5

—Pulsar2 Link, How to Convert LLM from Huggingface to axmodel

—WaveMatrix NPU HOST LLM Runtime

Support Platform

—WM9955
—WM9955 DEMO Board
—M.2 Accelerator card

Image Process | Chips | input size | image num | ViT encoder | TTFT (640 tokens) | Decode speed | CMM | Flash | |--|--|--|--|--|--|--|--| | WM9955 | 576x768 | 1 | 1685.554 ms | 361.8 ms | 44.6 tokens/sec | TBD | TBD |

How to use

Install axllm

shell

git clone -b axllm https://github.com/WaveMatrix/ax-llm.git
cd ax-llm
./install.sh

Model Download (Hugging Face)

Create directory then enter:

shell

mkdir -p WaveMatrix/PaddleOCR-VL-1.5
cd WaveMatrix/PaddleOCR-VL-1.5
hf download WaveMatrix/PaddleOCR-VL-1.5 --local-dir .

# Content
tree -L 1
.
|-- README.md
|-- assets
|-- config.json
|-- model.embed_tokens.weight.bfloat16.bin
|-- paddleocr_vl_p128_l0_together.axmodel
...
|-- paddleocr_vl_p128_l17_together.axmodel
|-- paddleocr_vl_post.axmodel
|-- post_config.json
|-- python
|-- tokenizer.model
|-- vision_cache
`-- vit_576x768.axmodel

Inference with WM9955 Host, such as WM9955 DEMO Board or M.2 card

Run the model via CLI

shell

axllm run WaveMatrix/PaddleOCR-VL-1.5/

After prompt, image >> will pop up：

—Just press "Enter" will go into "text only" mode
—Or input the path of the image to start the OCR process

Example for image mode:

[image]

shell

root@WM9955 # axllm run WaveMatrix/PaddleOCR-VL-1.5/
20:45:21.515 INF Init:218 | LLM init start
tokenizer_type = 0
 95% | ##############################   |  20 /  21 [1.97s<2.07s, 10.13 count/s] init post axmodel ok,remain_cmm(4330 MB)
20:45:23.490 INF Init:368 | max_token_len : 2047
20:45:23.490 INF Init:371 | kv_cache_size : 256, kv_cache_num: 2047
20:45:23.490 INF Init:374 | prefill_token_num : 128
20:45:23.490 INF Init:379 | grp: 1, prefill_max_kv_cache_num : 1
20:45:23.490 INF Init:379 | grp: 2, prefill_max_kv_cache_num : 128
20:45:23.490 INF Init:379 | grp: 3, prefill_max_kv_cache_num : 256
20:45:23.490 INF Init:379 | grp: 4, prefill_max_kv_cache_num : 384
20:45:23.490 INF Init:379 | grp: 5, prefill_max_kv_cache_num : 512
20:45:23.490 INF Init:379 | grp: 6, prefill_max_kv_cache_num : 640
20:45:23.490 INF Init:384 | prefill_max_token_num : 640
20:45:23.490 INF Init:27 | LLaMaEmbedSelector use mmap
100% | ################################ |  21 /  21 [1.98s<1.98s, 10.62 count/s] embed_selector init ok
20:45:24.028 INF Init:453 | PaddleOCRVL: encoder input nSize=5334336 -> eff_nSize=1333584 (float32 input)
20:45:24.029 WRN Init:469 | Qwen-VL vision size override: cfg=448x448 bytes=602112, model_input_bytes=5334336 -> 756x588 (factor-search).
20:45:24.029 INF Init:661 | PaddleOCR-VL token ids: vision_start=101305 image_pad=100295 video_pad=100295
20:45:24.029 INF Init:686 | VisionModule init ok: type=PaddleOCRVL, tokens_per_block=567, embed_size=1024, out_dtype=fp32
20:45:24.029 WRN Init:695 | Vision preprocess backend: SimpleCV (OpenCV not found at build time; minor differences vs OpenCV are possible)
20:45:24.031 INF load_config:282 | load config:
20:45:24.031 INF load_config:282 | {
20:45:24.031 INF load_config:282 |     "enable_repetition_penalty": false,
20:45:24.031 INF load_config:282 |     "enable_temperature": false,
20:45:24.031 INF load_config:282 |     "enable_top_k_sampling": false,
20:45:24.031 INF load_config:282 |     "enable_top_p_sampling": false,
20:45:24.031 INF load_config:282 |     "penalty_window": 20,
20:45:24.031 INF load_config:282 |     "repetition_penalty": 1.0,
20:45:24.031 INF load_config:282 |     "temperature": 0.6,
20:45:24.031 INF load_config:282 |     "top_k": 1,
20:45:24.031 INF load_config:282 |     "top_p": 0.9
20:45:24.031 INF load_config:282 | }
20:45:24.031 INF Init:448 | LLM init ok
Commands:
  /q, /exit  Exit
  /reset     Reset kvcache
  /dd        Delete a round of conversation
  /pp        Print history
Ctrl+C: Stop the current ongoing generation
VLM enabled: after each prompt, input image path (empty = text-only). Use "video:<frames_dir>" for video.
----------------------------------------
prompt >> OCR:
image >> /Path/To/Your/WaveMatrix/PaddleOCR-VL-1.5/assets/IMG_0462.JPG
20:45:30.031 INF EncodeForContent:1058 | PaddleOCRVL pixel_values bytes=1333584 min=0 max=255 (w=756 h=588 ps=14)
20:45:31.726 INF EncodeForContent:1102 | vision cache store: /Path/To/Your/WaveMatrix/PaddleOCR-VL-1.5/assets/IMG_0462.JPG
20:45:31.760 INF SetKVCache:747 | prefill_grpid:6 kv_cache_num:640 precompute_len:0 input_num_token:596
20:45:31.760 INF SetKVCache:749 | current prefill_max_token_num:640
20:45:31.760 INF SetKVCache:752 | first run
20:45:31.761 INF Run:805 | input token num : 596, prefill_split_num : 5
20:45:31.761 INF Run:845 | prefill chunk p=0 history_len=0 grpid=1 kv_cache_num=0 input_tokens=128
20:45:31.761 INF Run:868 | prefill indices shape: p=0 idx_elems=384 idx_rows=3 pos_rows=3
20:45:31.815 INF Run:845 | prefill chunk p=1 history_len=128 grpid=2 kv_cache_num=128 input_tokens=128
20:45:31.816 INF Run:868 | prefill indices shape: p=1 idx_elems=384 idx_rows=3 pos_rows=3
20:45:31.873 INF Run:845 | prefill chunk p=2 history_len=256 grpid=3 kv_cache_num=256 input_tokens=128
20:45:31.873 INF Run:868 | prefill indices shape: p=2 idx_elems=384 idx_rows=3 pos_rows=3
20:45:31.937 INF Run:845 | prefill chunk p=3 history_len=384 grpid=4 kv_cache_num=384 input_tokens=128
20:45:31.937 INF Run:868 | prefill indices shape: p=3 idx_elems=384 idx_rows=3 pos_rows=3
20:45:32.006 INF Run:845 | prefill chunk p=4 history_len=512 grpid=5 kv_cache_num=512 input_tokens=84
20:45:32.006 INF Run:868 | prefill indices shape: p=4 idx_elems=384 idx_rows=3 pos_rows=3
20:45:32.088 INF Run:1010 | ttft: 327.20 ms
James Landay-VR
14175

20:45:32.374 NTC Run:1132 | hit eos,avg 38.47 token/s
20:45:32.374 INF GetKVCache:721 | precompute_len:597, remaining:43

Start the service (OpenAI API compatiable)

axllm serve WaveMatrix/PaddleOCR-VL-1.5/

Example journal:

shell

root@WM9955 # axllm serve WaveMatrix/PaddleOCR-VL-1.5/
20:47:54.027 INF Init:218 | LLM init start
tokenizer_type = 0
 95% | ##############################   |  20 /  21 [1.96s<2.05s, 10.22 count/s] init post axmodel ok,remain_cmm(4330 MB)
20:47:55.983 INF Init:368 | max_token_len : 2047
20:47:55.983 INF Init:371 | kv_cache_size : 256, kv_cache_num: 2047
20:47:55.983 INF Init:374 | prefill_token_num : 128
20:47:55.983 INF Init:379 | grp: 1, prefill_max_kv_cache_num : 1
20:47:55.983 INF Init:379 | grp: 2, prefill_max_kv_cache_num : 128
20:47:55.983 INF Init:379 | grp: 3, prefill_max_kv_cache_num : 256
20:47:55.983 INF Init:379 | grp: 4, prefill_max_kv_cache_num : 384
20:47:55.983 INF Init:379 | grp: 5, prefill_max_kv_cache_num : 512
20:47:55.983 INF Init:379 | grp: 6, prefill_max_kv_cache_num : 640
20:47:55.983 INF Init:384 | prefill_max_token_num : 640
20:47:55.983 INF Init:27 | LLaMaEmbedSelector use mmap
100% | ################################ |  21 /  21 [1.96s<1.96s, 10.72 count/s] embed_selector init ok
20:47:56.526 INF Init:453 | PaddleOCRVL: encoder input nSize=5334336 -> eff_nSize=1333584 (float32 input)
20:47:56.526 WRN Init:469 | Qwen-VL vision size override: cfg=448x448 bytes=602112, model_input_bytes=5334336 -> 756x588 (factor-search).
20:47:56.526 INF Init:661 | PaddleOCR-VL token ids: vision_start=101305 image_pad=100295 video_pad=100295
20:47:56.526 INF Init:686 | VisionModule init ok: type=PaddleOCRVL, tokens_per_block=567, embed_size=1024, out_dtype=fp32
20:47:56.526 WRN Init:695 | Vision preprocess backend: SimpleCV (OpenCV not found at build time; minor differences vs OpenCV are possible)
20:47:56.528 INF load_config:282 | load config:
20:47:56.528 INF load_config:282 | {
20:47:56.528 INF load_config:282 |     "enable_repetition_penalty": false,
20:47:56.528 INF load_config:282 |     "enable_temperature": false,
20:47:56.528 INF load_config:282 |     "enable_top_k_sampling": false,
20:47:56.528 INF load_config:282 |     "enable_top_p_sampling": false,
20:47:56.528 INF load_config:282 |     "penalty_window": 20,
20:47:56.528 INF load_config:282 |     "repetition_penalty": 1.0,
20:47:56.528 INF load_config:282 |     "temperature": 0.6,
20:47:56.528 INF load_config:282 |     "top_k": 1,
20:47:56.528 INF load_config:282 |     "top_p": 0.9
20:47:56.528 INF load_config:282 | }
20:47:56.528 INF Init:448 | LLM init ok
Starting server on port 8000 with model 'WaveMatrix/PaddleOCR-VL-1.5'...
API URLs:
  GET  http://127.0.0.1:8000/health
  GET  http://127.0.0.1:8000/v1/models
  POST http://127.0.0.1:8000/v1/chat/completions
  GET  http://10.168.232.217:8000/health
  GET  http://10.168.232.217:8000/v1/models
  POST http://10.168.232.217:8000/v1/chat/completions
  GET  http://172.17.0.1:8000/health
  GET  http://172.17.0.1:8000/v1/models
  POST http://172.17.0.1:8000/v1/chat/completions
Aliases:
  GET  http://127.0.0.1:8000/models
  POST http://127.0.0.1:8000/chat/completions
  GET  http://10.168.232.217:8000/models
  POST http://10.168.232.217:8000/chat/completions
  GET  http://172.17.0.1:8000/models
  POST http://172.17.0.1:8000/chat/completions
OpenAI API Server starting on http://0.0.0.0:8000
Max concurrency: 1
Models: WaveMatrix/PaddleOCR-VL-1.5

Python Inference (Optional)

Python inference script is under python/

shell

python3 python/infer_axmodel.py \
  --hf_model ./python/paddleocr_vl_1-5_tokenizer \
  --axmodel_path . \
  --vit_model_path ./vit_576x768.axmodel \
  --image_path ./assets/IMG_0462.JPG \
  --task ocr

Task type supported(--task):

—ocr - General text recognition
—table - Table recognition
—chart - Chart recognition
—formula - Formula cognition
—spotting - Text spotting
—seal - Seal recognition

End-to-End Metrics (WM9955N)

Metric	Value
Max TTFT (640 tokens)	361.8 ms
Decode speed	44.6 tokens/s
ViT latency (576x768)	1685.554 ms

Subgraph Latency

Stage	Subgraph	Latency
Prefill	g1	2.551 ms
Prefill	g2	2.883 ms
Prefill	g3	3.158 ms
Prefill	g4	3.413 ms
Prefill	g5	3.795 ms
Prefill	g6	4.007 ms
Decode	g0	0.949 ms
Post-process	-	5.313 ms
ViT	-	1685.554 ms

PaddleOCR-VL-1.5

Convert tools links:

Support Platform

How to use

Install axllm

Model Download (Hugging Face)

Inference with WM9955 Host, such as WM9955 DEMO Board or M.2 card

Run the model via CLI

Start the service (OpenAI API compatiable)

Python Inference (Optional)

End-to-End Metrics (WM9955N)

Subgraph Latency

Discussion