CoolFace
Modelpublic

mlx-community/GOT-OCR2_0-8bit

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face
0likes140downloads
Model Card

mlx-community/GOT-OCR2_0-8bit

8-bit MLX quantization of `stepfun-ai/GOT-OCR2_0`, the 560M-parameter OCR-2.0 model. Converted with mlx-vlm 0.6.14 (mlx 0.32.0) for inference on Apple Silicon.

Fidelity was measured against the unquantized bf16 source and against the reference PyTorch implementation, not assumed. Every number below comes from a run recorded in this card.

Requires `mlx-vlm` with GOT-OCR 2.0 support, which is currently open as Blaizzy/mlx-vlm#1908. The PR is not merged yet, so a released mlx-vlm cannot load this repo.
bash
python -m mlx_vlm generate \
    --model mlx-community/GOT-OCR2_0-8bit \
    --image document.png \
    --prompt "OCR: " \
    --max-tokens 1024

GOT is not a chat model. It takes two instructions: OCR: for plain text and OCR with format: for structured output (tables, formulas, sheet music). Any other prompt is out of distribution.

Quantization

Bits8
Group size64
Modeaffine
Effective bits/weight9.851
Size on disk689 MB
Quantized tensors169 language-model tensors
Vision towerbf16, unquantized (0 .scales under vision_tower)
Projectorbf16, unquantized

The effective figure sits well above the nominal 8 because mlx-vlm quantizes only the language model. The vision tower and the projector are 96.7M of the 560.5M parameters, 17%, and stay bf16 by design. Packing checks out exactly: U32 x (32/8) = 463,785,984, the quantized parameter count.

Tied embeddings are 155.5M of those 463.8M, 34%, much higher than in a typical conversion where the transformer body dominates. They are the worst-hit tensor in both quantizations, which is what the per-tensor numbers below show.

Fidelity

Weight level, against the bf16 source

Weight SNR42.67 dB
Tensors compared169
Parameters compared463,785,984
Worst tensorembed_tokens (41.07 dB)

Exact, prompt-free and hardware-independent: the quantized weights are dequantized and compared element-wise against the source.

Against the reference PyTorch implementation

One forward pass, same image and same prompt (the MPT conversation the model was trained with), comparing the final-position logits over all 151,860 classes.

Comparisonmax abs errorcosineargmax
MLX bf16 vs torch fp320.474690.99979109same, top-5 identical
MLX fp32 vs torch fp320.0001301.0000000000same

The second row is the one that matters: the source weights are bf16, so running the MLX port in fp32 loses nothing, and the residual is porting error alone. The first row's larger figure is bf16 rounding, not a defect.

Task level, against ground truth

Six documents rendered locally with exactly known text (invoice, lab report, shipping label, receipt, spec table, rotated receipt), transcribed with OCR: and scored directly. This sidesteps using bf16 as the reference at all.

VariantfieldcontentnumericCER vs bf16tok/speak GB
bf160.86840.96050.97200 (ref)138.32.50
8-bit0.86840.96050.97200.0000210.82.06
4-bit0.89470.94740.96230.0116272.91.83

field counts required strings present exactly, content ignores markup and whitespace, numeric counts ground-truth numbers recovered.

The 8-bit output is byte-identical to bf16 on all six documents, down to the same fields missed.

Two caveats a reader should have:

  • —The 4-bit `field` score is higher than bf16 and that is not an improvement. The denominator is 38 strings, so one string is 0.026. 4-bit happens to get the invoice fully right and then loses 12.00 on the receipt. content and numeric, which are less format-sensitive, both move the other way. Treat the three variants as separated by one or two strings, not by field.
  • —`shipping_label` loses the same three fields in every variant, bf16 included (1Z-994-AX-77310582, Halberd Components, 3011AB). That is a limit of the source model on that document, not a quantization effect.

Throughput measured on an M-series Mac during the same run, single image, generation only.

What was not measured

  • —Only the plain 1024x1024 single-crop path. GOT's fine-grained mode (region by box or by colour), its multi-crop path for dense pages, and its multi-page mode are not exercised by anything above.
  • —`OCR with format: ` was not scored. The accuracy table uses OCR: only. Format mode was smoke-tested and runs, but no ground-truth scoring was done for tables, formulas or sheet music.
  • —No perplexity, top-1 agreement or KL. GOT only emits transcriptions, so generic passages give meaningless perplexity. The ground-truth OCR scoring above replaces that layer rather than supplementing it.
  • —Six synthetic documents, one language. They are rendered, not photographed, and English only. Real scans, handwriting and the multilingual claims of the source are untested here.
  • —No accuracy measurement at long context. Everything ran well under the 32k window.

Related

Source model: `stepfun-ai/GOT-OCR2_0` (paper).