CoolFace
Modelpublic

mazenDDr/receipt-vlm-qwen2.5-vl-3b-awq

sourceHugging Faceapache-2.0updated 11d agoView on Hugging Face
0likes21downloads
Model Card

Receipt field extraction — AWQ W4A16

The fine-tuned model quantized to 4-bit with AWQ (W4A16, language model only; the vision tower stays in bf16). 3.39 GB, down from 7.51 GB.

This is the build the project recommends for a GPU server: smallest, fastest, and its accuracy is within noise of the bf16 model it came from.

Field F1Latency p50Output tok/s
Fine-tuned bf160.8522.07 s51.1
This build0.8471.05 s103.1

Paired against its own bf16 source on the same 100 receipts: −0.005 [−0.026, +0.013]. The interval includes zero, so the difference is not real. Calibrated on 256 training receipts with llm-compressor.

Use it with vLLM

bash
vllm serve mazenDDr/receipt-vlm-qwen2.5-vl-3b-awq --max-model-len 4096 --limit-mm-per-prompt '{"image": 1}'

Serve this with vLLM, not transformers. At the time it was produced, transformers 5.14 with compressed-tensors 0.18 could not load this checkpoint (two reader bugs in the packed-quantized format), and would have decompressed it to bf16 anyway — losing the point of the 4-bit weights. vLLM's W4A16 kernels run it natively.

How it was trained

QLoRA on one RTX 5060 Ti (16 GB): the base model in 4-bit NF4 with the vision tower kept in bf16, LoRA rank 16 / alpha 32 on the language-model layers only, learning rate 4e-4, 2 epochs, loss on the answer tokens alone. 1.07 h, 7.6 GB peak VRAM, 29.9 M trainable parameters.

Training data is CORD v2 (CC BY 4.0), split 773 / 99 / 100 after dropping 28 images that appeared in more than one split under a 256-bit perceptual hash. Every choice was made on the dev split; the test split was scored once.

Limitations

  • —It is trained on one dataset's conventions. CORD labels transcribe what is printed, stray marks included, and Indonesian receipts use . and , interchangeably as thousands separators. The model learned those conventions; on receipts from elsewhere, formatting may not match your expectations.
  • —The remaining errors are structural, not perceptual. All 100 held-out receipts produced valid JSON. The worst one has every value correct and every key wrong. Separator conventions alone account for about a third of the numeric gap.
  • —Not for accounting without review. It misfiles fields often enough that a human should check anything that matters financially.

Links

  • —Code, evaluation harness and write-ups: <https://github.com/mazenDDr/receipt-vlm>
  • —Field guide, one-receipt tour, and 100 recorded extractions: <https://mazenddr.github.io/receipt-vlm/>

Citation

Receipts from CORD v2 (Park et al., CORD: A Consolidated Receipt Dataset for Post-OCR Parsing, 2019), CC BY 4.0. Base model Qwen/Qwen2.5-VL-3B-Instruct, Apache-2.0.