SanjeevSOLANKI/PaddleOCR-VL-1.6-GGUF

PaddleOCR-VL-1.6 — GGUF quants for llama.cpp

GGUF quantizations of [PaddlePaddle/PaddleOCR-VL](https://huggingface.co/PaddlePaddle/PaddleOCR-VL) (the 1.6 release) — a ~0.9 B vision-language OCR model (NaViT encoder + ERNIE-4.5-0.3B) — for running document OCR locally with llama.cpp.

These are community quants. The original model and weights are © PaddlePaddle, released under Apache-2.0; this repo redistributes derivative (quantized) copies under the same license, with attribution. No model architecture or behavior was changed — only weight precision.

Quick start

👉 Use `PaddleOCR-VL-1.6-Q4_K_M.gguf` (the recommended file) + the required `mmproj`.

# 1. download the recommended quant + the vision projector (required)
huggingface-cli download SanjeevSOLANKI/PaddleOCR-VL-1.6-GGUF \
  PaddleOCR-VL-1.6-Q4_K_M.gguf PaddleOCR-VL-1.6-mmproj.gguf --local-dir .

# 2. serve it (8 GB GPU → ~1.6 GB VRAM, 8-way parallel)
llama-server -m PaddleOCR-VL-1.6-Q4_K_M.gguf \
  --mmproj PaddleOCR-VL-1.6-mmproj.gguf -c 8192 -np 8 -ngl 99 --port 8080

Then POST a page image with the text prompt `OCR:` (OpenAI vision format) to http://localhost:8080/v1/chat/completions. Prefer higher-precision quants? Use Q5_K_M / Q6_K / Q8_0 instead — same mmproj. See Validation and the ⚠️ repetition-loop note before production use.

Files

The F16 reference is not re-hosted here (it's identical to upstream) — get it from the official GGUF repo. It's used below only as the comparison baseline.

Validation & findings

Why not perplexity / KL-divergence?

The usual quant metric (KL-divergence or perplexity vs F16 on a text corpus) does not apply to this model. PaddleOCR-VL is a vision OCR model — given plain text without an image, it's out-of-distribution: measured F16 perplexity on academic text was ~160 (a real text-LM is ~5–30), and in that near-random regime the comparison is meaningless (Q8_0 even scored a lower perplexity than F16). So we validated on the actual OCR task instead.

Method (apple-to-apple)

All five models (F16 + four quants) OCR'd the same four pages (title/authors, dense body text, equation+table, references) with identical, deterministic settings (-np 1, temperature 0). At -np 1 the model is fully deterministic — F16-vs-F16 = 100% identical — so any quant divergence is real, not run-to-run noise.

Results (vs F16)

Interpretation (the important part)

• —Every quant extracts the same values as F16. Each win-rate / conflict-rate / fraction on the table page (51.3%, 46.3%, 37 / 80, 82.5%, …) is reproduced correctly by all of Q4–Q8 — verified directly, not assumed.
• —The ~30% text divergence is cosmetic, not error. It's reading-order, whitespace, and where each transcription starts/ends — and it's non-monotonic in precision (Q4 ≈ Q8). If it were graded quantization loss, Q8 would track F16 most closely; it doesn't. That signature means the divergence is generation-path sensitivity, not accuracy loss.
• —Conclusion: no quant in Q4–Q8 shows OCR value-level degradation vs F16. Q4_K_M is the recommended default (smallest, fastest, values intact). Q5–Q8 are offered for headroom but showed no measurable accuracy advantage on this task.

⚠️ Known failure mode — repetition loops. On one of the four test pages (a highly repetitive template layout), the bare model degenerated into a runaway loop — repeating a line hundreds of times. Critically, this happened to F16 and several quants alike (F16 was the worst; Q8 didn't loop; Q4 did) — there is no precision ordering, so it is a model/inference limitation, not a quantization defect, and it occurred even with `repeat_penalty 1.15`. See Limitations for mitigations. The "values correct" finding above applies to well-behaved pages; repetition-prone layouts can break for any precision including F16.

Honest limitations

Sanity-grade, not a benchmark: 4 pages, one document, English, accuracy checked against F16 + targeted value spot-checks, not against human ground-truth labels (e.g. OmniDocBench). It establishes "Q4 ≈ F16 on values for well-formed pages" — it is not a multilingual or large-corpus accuracy claim, and (see warning above) it does not mean every page transcribes cleanly. For rigorous accuracy numbers, see the upstream model's OmniDocBench results.

Performance (21-page paper, RTX A4000 8 GB, llama.cpp, 8-way parallel)

Q4 is ~1.7× faster than F16 and lighter, because decode (the bottleneck, ~89% of runtime) reads the weights for every generated token — 4-bit weights = 1/3 the memory traffic.

Usage (llama.cpp)

llama-server -m PaddleOCR-VL-1.6-Q4_K_M.gguf \
  --mmproj PaddleOCR-VL-1.6-mmproj.gguf \
  -c 8192 -np 8 -ngl 99 --port 8080

Then send a page image with the prompt `OCR:` (OpenAI vision format). On an 8 GB GPU this runs at ~1.6 GB VRAM and 8-way parallel.

Tips

• —This is a specialized OCR model, not a general chat VLM. Use the trained prompt OCR:; custom instructions ("convert to markdown…") make it worse.
• —Repetition loops are a real failure mode (affects full-precision F16 too — not a quantization issue): on highly repetitive layouts the model can run away repeating a line. "repeat_penalty": 1.15 is not always enough. Stronger mitigations: raise repeat_penalty (≈1.3) and/or enable DRY sampling (--dry-multiplier 0.8), cap max_tokens, and detect runaway output and retry/flag the page. For repetition-heavy documents, prefer the official PaddleOCR-VL pipeline (its layout stage avoids feeding whole repetitive pages to the VLM) or a model with a built-in no-repeat-ngram processor.
• —Output is flat text + LaTeX (the bare VLM). The structured layout (HTML tables, headings, figure crops) comes from PaddleOCR's separate layout stage, which is not in the GGUF — for fully-structured Markdown use the official PaddleOCR-VL pipeline.

Quantized with

llama-quantize (llama.cpp), static k-quants from the F16 GGUF. Imatrix was intentionally not used: at Q4–Q8 the values are preserved on the OCR task (see validation), and sub-4-bit IQ quants aren't useful for a 286 MB model whose constraint is decode speed, not size.

Test conditions

All quantization, validation, and benchmarks were run on a single machine. Numbers (speed/VRAM) are specific to this setup and will differ on yours.

License & attribution

Apache-2.0, inherited from the base model [PaddlePaddle/PaddleOCR-VL](https://huggingface.co/PaddlePaddle/PaddleOCR-VL). See that repo for the original LICENSE/NOTICE. "PaddleOCR" is a trademark of its owners; this is an unofficial community quantization and is not endorsed by PaddlePaddle.

Disclaimer

These are unofficial, community-produced quantizations, provided "as is" without warranty of any kind, express or implied, including fitness for a particular purpose. Use at your own risk.

• —Not affiliated with or endorsed by PaddlePaddle/Baidu. For the authoritative model, weights, and benchmarks, use the official repository.
• —Quantization is lossy. The validation here is sanity-grade (4 pages, one English document, checked against F16 + targeted value spot-checks — not against human ground-truth labels or a standard benchmark like OmniDocBench). It does not guarantee accuracy on your documents, languages, or layouts.
• —Do not rely on OCR output for safety-, legal-, medical-, or financial-critical decisions without independent human verification.
• —Benchmark numbers (speed, VRAM, timings) are specific to the hardware/software in Test conditions and will vary on other systems.
• —You are responsible for complying with the Apache-2.0 license terms and any applicable laws when using or redistributing these files.