octopusmegalopod/some-paddleocr1.5-vl-ggufs

GGUFs for use in llama.cpp. ~~As of b8110 a specific chat template needs to be used, else the model crashes. The chat template is included in the repo files.~~

The chat template is finnicky and should probably not be used. Instead, for now a special tag <__media__> should be added before the actual user prompt string. I can confirm this gives better results.

The base files are BF16.

Basic usage with llama-server: llama-server -m PaddleOCR-VL-1.5.gguf --mmproj mmproj-PaddleOCR-VL-1.5.gguf -c 20000 --fit off

Then make a request with an image attached and prompt e.g: <__media__>OCR:

PaddleOCR-VL is expected to be used with a separate Paddle layout analysis model (NOT included in llama.cpp - it's in the PaddlePaddle library). However, for simpler use cases PaddleOCR-VL can work alone.

So far on short multilingual line texts, it performs well on low quants (Q4KM lm & Q4_1 mmproj). Performance remains to be seen on more complex tasks.