Modelpublic
WaveMatrix/PaddleOCR-VL-1.5

sourceHugging Facebsd-3-clauseupdated 22d agoView on Hugging Face
0likes20downloads
README.md275 linesDownload Raw Back to root
1---2library_name: transformers3license: bsd-3-clause4base_model:5- PaddlePaddle/PaddleOCR-VL-1.56pipeline_tag: image-text-to-text7tags:8- PaddleOCR-VL-1.59- OCR10- VLM11language:12- zh13- en14---15 16# PaddleOCR-VL-1.517 18This version of PaddleOCR-VL-1.5 has been converted to run on the WaveMatrix NPU using **w4a16** quantization.19 20Compatible with Pulsar2 version: 5.021 22## Convert tools links:23 24For those who are interested in model conversion, you can try to export axmodel through the original repo:25 26- https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.527 28- [Pulsar2 Link, How to Convert LLM from Huggingface to axmodel](https://pulsar2-docs.readthedocs.io/en/latest/appendix/build_llm.html)29 30- [WaveMatrix NPU HOST LLM Runtime](https://github.com/WaveMatrix/ax-llm)31 32## Support Platform33 34- WM995535  - WM9955 DEMO Board36  - [M.2 Accelerator card](https://docs.m5stack.com/zh_CN/ai_hardware/LLM-8850_Card)37 38**Image Process**39| Chips | input size | image num | ViT encoder | TTFT (640 tokens) | Decode speed | CMM | Flash |40|--|--|--|--|--|--|--|--|41| WM9955 | 576x768 | 1 | 1685.554 ms | 361.8 ms | 44.6 tokens/sec | TBD | TBD |42 43## How to use44 45## Install axllm46 47```shell48git clone -b axllm https://github.com/WaveMatrix/ax-llm.git49cd ax-llm50./install.sh51```52 53## Model Download (Hugging Face)54Create directory then enter:55 56```shell57mkdir -p WaveMatrix/PaddleOCR-VL-1.558cd WaveMatrix/PaddleOCR-VL-1.559hf download WaveMatrix/PaddleOCR-VL-1.5 --local-dir .60 61# Content62tree -L 163.64|-- README.md65|-- assets66|-- config.json67|-- model.embed_tokens.weight.bfloat16.bin68|-- paddleocr_vl_p128_l0_together.axmodel69...70|-- paddleocr_vl_p128_l17_together.axmodel71|-- paddleocr_vl_post.axmodel72|-- post_config.json73|-- python74|-- tokenizer.model75|-- vision_cache76`-- vit_576x768.axmodel77```78 79## Inference with WM9955 Host, such as WM9955 DEMO Board or M.2 card80 81### Run the model via CLI82 83```shell84axllm run WaveMatrix/PaddleOCR-VL-1.5/85```86 87After `prompt`, `image >>` will pop up：88- Just press "Enter" will go into "text only" mode89- Or input the path of the image to start the OCR process90 91Example for image mode:92 93![TestImage](assets/IMG_0462.JPG)94 95```shell96root@WM9955 # axllm run WaveMatrix/PaddleOCR-VL-1.5/9720:45:21.515 INF Init:218 | LLM init start98tokenizer_type = 099 95% | ##############################   |  20 /  21 [1.97s<2.07s, 10.13 count/s] init post axmodel ok,remain_cmm(4330 MB)10020:45:23.490 INF Init:368 | max_token_len : 204710120:45:23.490 INF Init:371 | kv_cache_size : 256, kv_cache_num: 204710220:45:23.490 INF Init:374 | prefill_token_num : 12810320:45:23.490 INF Init:379 | grp: 1, prefill_max_kv_cache_num : 110420:45:23.490 INF Init:379 | grp: 2, prefill_max_kv_cache_num : 12810520:45:23.490 INF Init:379 | grp: 3, prefill_max_kv_cache_num : 25610620:45:23.490 INF Init:379 | grp: 4, prefill_max_kv_cache_num : 38410720:45:23.490 INF Init:379 | grp: 5, prefill_max_kv_cache_num : 51210820:45:23.490 INF Init:379 | grp: 6, prefill_max_kv_cache_num : 64010920:45:23.490 INF Init:384 | prefill_max_token_num : 64011020:45:23.490 INF Init:27 | LLaMaEmbedSelector use mmap111100% | ################################ |  21 /  21 [1.98s<1.98s, 10.62 count/s] embed_selector init ok11220:45:24.028 INF Init:453 | PaddleOCRVL: encoder input nSize=5334336 -> eff_nSize=1333584 (float32 input)11320:45:24.029 WRN Init:469 | Qwen-VL vision size override: cfg=448x448 bytes=602112, model_input_bytes=5334336 -> 756x588 (factor-search).11420:45:24.029 INF Init:661 | PaddleOCR-VL token ids: vision_start=101305 image_pad=100295 video_pad=10029511520:45:24.029 INF Init:686 | VisionModule init ok: type=PaddleOCRVL, tokens_per_block=567, embed_size=1024, out_dtype=fp3211620:45:24.029 WRN Init:695 | Vision preprocess backend: SimpleCV (OpenCV not found at build time; minor differences vs OpenCV are possible)11720:45:24.031 INF load_config:282 | load config:11820:45:24.031 INF load_config:282 | {11920:45:24.031 INF load_config:282 |     "enable_repetition_penalty": false,12020:45:24.031 INF load_config:282 |     "enable_temperature": false,12120:45:24.031 INF load_config:282 |     "enable_top_k_sampling": false,12220:45:24.031 INF load_config:282 |     "enable_top_p_sampling": false,12320:45:24.031 INF load_config:282 |     "penalty_window": 20,12420:45:24.031 INF load_config:282 |     "repetition_penalty": 1.0,12520:45:24.031 INF load_config:282 |     "temperature": 0.6,12620:45:24.031 INF load_config:282 |     "top_k": 1,12720:45:24.031 INF load_config:282 |     "top_p": 0.912820:45:24.031 INF load_config:282 | }12920:45:24.031 INF Init:448 | LLM init ok130Commands:131  /q, /exit  Exit132  /reset     Reset kvcache133  /dd        Delete a round of conversation134  /pp        Print history135Ctrl+C: Stop the current ongoing generation136VLM enabled: after each prompt, input image path (empty = text-only). Use "video:<frames_dir>" for video.137----------------------------------------138prompt >> OCR:139image >> /Path/To/Your/WaveMatrix/PaddleOCR-VL-1.5/assets/IMG_0462.JPG14020:45:30.031 INF EncodeForContent:1058 | PaddleOCRVL pixel_values bytes=1333584 min=0 max=255 (w=756 h=588 ps=14)14120:45:31.726 INF EncodeForContent:1102 | vision cache store: /Path/To/Your/WaveMatrix/PaddleOCR-VL-1.5/assets/IMG_0462.JPG14220:45:31.760 INF SetKVCache:747 | prefill_grpid:6 kv_cache_num:640 precompute_len:0 input_num_token:59614320:45:31.760 INF SetKVCache:749 | current prefill_max_token_num:64014420:45:31.760 INF SetKVCache:752 | first run14520:45:31.761 INF Run:805 | input token num : 596, prefill_split_num : 514620:45:31.761 INF Run:845 | prefill chunk p=0 history_len=0 grpid=1 kv_cache_num=0 input_tokens=12814720:45:31.761 INF Run:868 | prefill indices shape: p=0 idx_elems=384 idx_rows=3 pos_rows=314820:45:31.815 INF Run:845 | prefill chunk p=1 history_len=128 grpid=2 kv_cache_num=128 input_tokens=12814920:45:31.816 INF Run:868 | prefill indices shape: p=1 idx_elems=384 idx_rows=3 pos_rows=315020:45:31.873 INF Run:845 | prefill chunk p=2 history_len=256 grpid=3 kv_cache_num=256 input_tokens=12815120:45:31.873 INF Run:868 | prefill indices shape: p=2 idx_elems=384 idx_rows=3 pos_rows=315220:45:31.937 INF Run:845 | prefill chunk p=3 history_len=384 grpid=4 kv_cache_num=384 input_tokens=12815320:45:31.937 INF Run:868 | prefill indices shape: p=3 idx_elems=384 idx_rows=3 pos_rows=315420:45:32.006 INF Run:845 | prefill chunk p=4 history_len=512 grpid=5 kv_cache_num=512 input_tokens=8415520:45:32.006 INF Run:868 | prefill indices shape: p=4 idx_elems=384 idx_rows=3 pos_rows=315620:45:32.088 INF Run:1010 | ttft: 327.20 ms157James Landay-VR15814175159 16020:45:32.374 NTC Run:1132 | hit eos,avg 38.47 token/s16120:45:32.374 INF GetKVCache:721 | precompute_len:597, remaining:43162```163 164### Start the service (OpenAI API compatiable)165 166```sh167axllm serve WaveMatrix/PaddleOCR-VL-1.5/168```169 170Example journal:171 172```shell173root@WM9955 # axllm serve WaveMatrix/PaddleOCR-VL-1.5/17420:47:54.027 INF Init:218 | LLM init start175tokenizer_type = 0176 95% | ##############################   |  20 /  21 [1.96s<2.05s, 10.22 count/s] init post axmodel ok,remain_cmm(4330 MB)17720:47:55.983 INF Init:368 | max_token_len : 204717820:47:55.983 INF Init:371 | kv_cache_size : 256, kv_cache_num: 204717920:47:55.983 INF Init:374 | prefill_token_num : 12818020:47:55.983 INF Init:379 | grp: 1, prefill_max_kv_cache_num : 118120:47:55.983 INF Init:379 | grp: 2, prefill_max_kv_cache_num : 12818220:47:55.983 INF Init:379 | grp: 3, prefill_max_kv_cache_num : 25618320:47:55.983 INF Init:379 | grp: 4, prefill_max_kv_cache_num : 38418420:47:55.983 INF Init:379 | grp: 5, prefill_max_kv_cache_num : 51218520:47:55.983 INF Init:379 | grp: 6, prefill_max_kv_cache_num : 64018620:47:55.983 INF Init:384 | prefill_max_token_num : 64018720:47:55.983 INF Init:27 | LLaMaEmbedSelector use mmap188100% | ################################ |  21 /  21 [1.96s<1.96s, 10.72 count/s] embed_selector init ok18920:47:56.526 INF Init:453 | PaddleOCRVL: encoder input nSize=5334336 -> eff_nSize=1333584 (float32 input)19020:47:56.526 WRN Init:469 | Qwen-VL vision size override: cfg=448x448 bytes=602112, model_input_bytes=5334336 -> 756x588 (factor-search).19120:47:56.526 INF Init:661 | PaddleOCR-VL token ids: vision_start=101305 image_pad=100295 video_pad=10029519220:47:56.526 INF Init:686 | VisionModule init ok: type=PaddleOCRVL, tokens_per_block=567, embed_size=1024, out_dtype=fp3219320:47:56.526 WRN Init:695 | Vision preprocess backend: SimpleCV (OpenCV not found at build time; minor differences vs OpenCV are possible)19420:47:56.528 INF load_config:282 | load config:19520:47:56.528 INF load_config:282 | {19620:47:56.528 INF load_config:282 |     "enable_repetition_penalty": false,19720:47:56.528 INF load_config:282 |     "enable_temperature": false,19820:47:56.528 INF load_config:282 |     "enable_top_k_sampling": false,19920:47:56.528 INF load_config:282 |     "enable_top_p_sampling": false,20020:47:56.528 INF load_config:282 |     "penalty_window": 20,20120:47:56.528 INF load_config:282 |     "repetition_penalty": 1.0,20220:47:56.528 INF load_config:282 |     "temperature": 0.6,20320:47:56.528 INF load_config:282 |     "top_k": 1,20420:47:56.528 INF load_config:282 |     "top_p": 0.920520:47:56.528 INF load_config:282 | }20620:47:56.528 INF Init:448 | LLM init ok207Starting server on port 8000 with model 'WaveMatrix/PaddleOCR-VL-1.5'...208API URLs:209  GET  http://127.0.0.1:8000/health210  GET  http://127.0.0.1:8000/v1/models211  POST http://127.0.0.1:8000/v1/chat/completions212  GET  http://10.168.232.217:8000/health213  GET  http://10.168.232.217:8000/v1/models214  POST http://10.168.232.217:8000/v1/chat/completions215  GET  http://172.17.0.1:8000/health216  GET  http://172.17.0.1:8000/v1/models217  POST http://172.17.0.1:8000/v1/chat/completions218Aliases:219  GET  http://127.0.0.1:8000/models220  POST http://127.0.0.1:8000/chat/completions221  GET  http://10.168.232.217:8000/models222  POST http://10.168.232.217:8000/chat/completions223  GET  http://172.17.0.1:8000/models224  POST http://172.17.0.1:8000/chat/completions225OpenAI API Server starting on http://0.0.0.0:8000226Max concurrency: 1227Models: WaveMatrix/PaddleOCR-VL-1.5228```229 230## Python Inference (Optional)231 232Python inference script is under `python/`233 234```shell235python3 python/infer_axmodel.py \236  --hf_model ./python/paddleocr_vl_1-5_tokenizer \237  --axmodel_path . \238  --vit_model_path ./vit_576x768.axmodel \239  --image_path ./assets/IMG_0462.JPG \240  --task ocr241```242 243Task type supported(`--task`):244- `ocr` - General text recognition245- `table` - Table recognition246- `chart` - Chart recognition247- `formula` - Formula cognition248- `spotting` - Text spotting249- `seal` - Seal recognition250 251## End-to-End Metrics (WM9955N)252 253| Metric | Value |254| -- | -- |255| Max TTFT (640 tokens) | 361.8 ms |256| Decode speed | 44.6 tokens/s |257| ViT latency (576x768) | 1685.554 ms |258 259### Subgraph Latency260 261| Stage | Subgraph | Latency |262| -- | -- | -- |263| Prefill | g1 | 2.551 ms |264| Prefill | g2 | 2.883 ms |265| Prefill | g3 | 3.158 ms |266| Prefill | g4 | 3.413 ms |267| Prefill | g5 | 3.795 ms |268| Prefill | g6 | 4.007 ms |269| Decode | g0 | 0.949 ms |270| Post-process | - | 5.313 ms |271| ViT | - | 1685.554 ms |272 273 274## Discussion275