ratlinghitman/qwen3vl-4b-receipt-extraction-gguf

Qwen3-VL-4B Receipt Extraction (GGUF Q4KM)

Reads a receipt image and returns it as structured JSON. GGUF build for llama.cpp, Ollama and LM Studio.

At a glance

Intended use

Turning a photo or scan of a retail receipt into structured JSON, for bookkeeping and expense pipelines, or as a starting point for research on document understanding.

Out of scope

Invoices, handwritten notes, and non-receipt images. The model was never trained on them. It also should not be trusted to make financial decisions on its own; treat the output as an extraction to be reviewed.

How to use

The model expects one image and this exact prompt, which is the prompt it was trained with:

Extract the receipt from the image into a structured JSON. Your output should contain ONLY correct JSON!

Two files: the quantized language model and its mmproj vision tower. You need both.

huggingface-cli download <your-hf-username>/qwen3vl-4b-receipt-extraction-gguf --local-dir gguf

llama-server -m gguf/*-Q4_K_M.gguf --mmproj gguf/mmproj-*-f16.gguf -ngl 99 -c 8192

Then query the OpenAI-compatible endpoint with an image and the prompt:

Extract the receipt from the image into a structured JSON. Your output should contain ONLY correct JSON!

Your llama.cpp build must support the Qwen3-VL qwen3vl_merger projector. LM Studio and Ollama load the same pair.

Output

A single JSON object with three top-level keys: menu (the line items), sub_total and total. Every value is a string, copied from the receipt.

{"menu": [{"nm": "Es Teh Manis", "cnt": "2", "price": "8,000"}],
 "sub_total": {"subtotal_price": "8,000"},
 "total": {"total_price": "8,000", "cashprice": "10,000", "changeprice": "2,000"}}

Evaluation

CORD-v2 test split, 100 receipts, greedy decoding, images capped at 1600 patches.

Measured in section 8.2b of finetune_qwen3vl.ipynb.

• —field_f1 flattens the predicted and gold JSON into (field-path, value) leaf pairs and scores precision/recall over them. It asks whether the right values landed at the right paths.
• —nTED (normalized tree-edit distance, following the CORD paper) compares the two JSON trees directly. It catches structural mistakes, like wrong nesting or a missing key, that leaf-flattening misses.

Neither subsumes the other, which is why both are here.

Training

Fine-tuned from Qwen/Qwen3-VL-4B-Instruct with LoRA on `naver-clova-ix/cord-v2` (800 train / 100 validation / 100 test receipts). Labels were losslessly normalized before training.

Training ran well past convergence on purpose, so there was a wide range of adapters to pick from. Validation loss and the task metrics peak at different times here: loss starts rising around epoch 3, while the best fieldf1 and nTED land at epoch 9 or later. The adapter published here is the one that scored best on the **validation** split by fieldf1, and it was then scored once on the held-out test split. Early-stopping on loss would have picked a noticeably worse one.

Quantization

The language model is quantized to Q4_K_M by llama.cpp's llama-quantize. This is data-free: no calibration set, no receipts involved. The vision tower is exported separately as an f16 mmproj file and is left unquantized.

It converts from the bf16 master rather than the GPTQ build, because llama.cpp cannot ingest GPTQ-packed weights. Being data-free, it gives up more than the calibrated GPTQ build does: field_f1 0.8737 here against 0.8840 for GPTQ, on the same test split.

Limitations

• —The test split is 100 receipts. Gaps of a point or two in field_f1 are inside the noise band and should be read as ties.
• —CORD-v2 is photographed Indonesian retail and restaurant receipts. Expect worse results on other layouts, languages, or document types (invoices, handwriting).
• —Field names mirror the raw CORD keys (nm, cnt, unitprice), not friendly names.
• —Values are transcribed text, not validated arithmetic. Nothing checks that the line items sum to the total. Do not use the output for financial decisions without review.
• —Schema-constrained decoding was tried and scored far worse than plain decoding (field_f1 falls roughly 35 points), so plain greedy decoding is what these numbers use and what is recommended.
• —Any timing or throughput figure depends heavily on hardware. Measure on the machine you intend to deploy on.

License

apache-2.0, inherited from the base model. The CORD-v2 dataset carries its own terms; see its dataset card.