doldolggi/EvoCUA-8B-20260105-GGUF

EvoCUA-8B GGUF — quantized and verified against click-coordinate regression

GGUF conversions of `meituan/EvoCUA-8B-20260105`, a Qwen3-VL-based computer-use agent (CUA).

What makes this repo different from other EvoCUA GGUFs: every quant here was measured against the original `transformers` model on a click-coordinate regression suite. A CUA's output is a click coordinate, not prose — a broken preprocessing step produces perfectly fluent text with systematically shifted coordinates. Existing GGUF repos ship quants with no evidence either way. This one ships the evidence, the harness, and the raw per-case results so you can re-run or extend it.

⚠️ Scope of the evidence. 16 grounding cases, English UI, single-turn, synthetic screenshots with constructed ground truth. No OSWorld end-to-end benchmark was run. Measured process/kernel noise floor is ~2 px — differences below that are not meaningful. Read the tables as "does quantization move the click", not as a task-success benchmark.

Results

All numbers below are 8B vs the original `transformers` model on the same 16 grounding cases. Two metrics are reported separately because they mean different things:

• —miss — the model emitted a coordinate that landed outside the target → grounding failure
• —non-click — the model chose wait/etc., so there is no coordinate → behavioural difference, not a failure

Where it actually breaks

IQ1 collapses completely — only 6 of 16 cases even produced a click, and 1 of those landed on target. Mean error is half a screen. The plain IQ1_M control collapses identically (5 misses either way), so this is the bit level, not the recipe. This tier doubles as a negative control: a harness where everything passes tells you nothing.

Above IQ1, on-target holds while precision decays monotonically — exact coordinate matches go 8 → 6 → 5 → 1 → 0 across Q5 → Q4 → Q3 → Q2 → IQ2.

⚠️ On-target holding is an artifact of our target sizes. Do not generalize it. Our targets are 46–232 px wide, and only one — a toggle — is as small as 46 px. Across the whole narrow set, error at Q2/IQ2 is 5–6 px mean but 20–25 px max. The one small target we do have tells the story best: that 46 px toggle stayed on target all the way down to IQ2, but its drift grew 0 → 1.0 → 2.2 → 6.7 → 7.2 px (Q5 → Q4 → Q3 → Q2 → IQ2) — at IQ2 it is spending a third of its half-width on drift. A 24 px icon is untested here and would have almost no margin left. "Q2 still hits 16/16" does not mean Q2 is safe to deploy.

Practical recommendation: Q4 is the floor for real UIs. Q3 only if you accept halved precision (max error 50 px+). Q2 and below are not recommended; IQ1 is unusable.

Unsloth Dynamic (UD) recipes: what actually helps

The UD method has two parts — a custom imatrix and a per-tensor bit allocation. We separated them. Because the 8B baseline quants already used an imatrix, we could hold the imatrix constant and vary only the recipe — a control the 32B run could not do.

• —The gain comes from the tensor allocation, not the imatrix. Holding the imatrix fixed and switching only the recipe drops max narrow error 16.0 → 4.1 px — above the ~2 px noise floor.
• —The imatrix source barely matters. Ours vs Unsloth's (computed on the base model, before the CUA fine-tune) tie at 8/15 exact. On 32B they were indistinguishable too. So using a base-model imatrix on a fine-tune is fine — and computing your own buys little.
• —⚠️ The UD advantage is tier-scoped. It is clear at Q4–Q5 and disappears at Q3/Q2 (UD-Q3 is slightly worse than Q3KM). "UD is always better" is not what this data says.
• —⚠️ IQ2_M was measured with the UD recipe only — no stock control at that tier.

Files

Q2 / IQ2 / IQ1 builds were measured but are not uploaded — the results say don't use them. Their per-case numbers are in results/ so you can check the claim.

32B sibling repo

32B weights live in their own repo: `doldolggi/EvoCUA-32B-20260105-GGUF` — same conversion pipeline, same harness, same author, measured against a BF16 baseline of that model. You will also find 32B measurements in this repo's `results/` (*_32b*.json), because several conclusions only hold when the two sizes are read against each other:

• —the BF16 baseline, the kernel-path control, and the 32K-context check were run on 32B;
• —the "UD helps" result was first seen on 32B and then confirmed here on 8B with a cleaner control (imatrix held constant — possible because the 8B baseline quants already used one);
• —a 32B observation that UD hurt at Q4 failed to replicate on 8B, so it was retracted as n=1 noise. You can only see that by having both.

The 8B repo carries the wider quantization ladder (Q5 → IQ1, including the collapse point); the 32B repo carries the larger model and its VRAM constraints.

Serving

llama-server -m evocua-8b-<quant>.gguf --mmproj mmproj-evocua-8b-f16.gguf \
  -ngl 99 -c 32768 --parallel 1 \
  --cache-type-k q8_0 --cache-type-v q8_0 -fa on \
  --reasoning-format none \
  --host 127.0.0.1 --port 8080

Traps that cost us time — please read

1. 1.`--reasoning-format none` is mandatory. With the default (auto), the thought block is moved into reasoning_content and the EvoCUA S2 parser — which reads <think> + <tool_call> out of content — silently sees nothing. No error, just an empty parse.
2. 2.Never pass `--reasoning-budget`. Truncating the CoT breaks the agent loop.
3. 3.4K screenshots get downscaled, unlike upstream. llama.cpp's Qwen3-VL branch hardcodes set_limit_image_tokens(8, 4096) and neither writes nor reads image_min_pixels / image_max_pixels in the mmproj GGUF. Cap = 4096 merged tokens = 4.19 MP. 1920×1080 → 1920×1088 = 2040 tokens, no downscale. 3840×2160 = 8.3 MP is downscaled. Pass --image-max-tokens 16384 if you feed 4K. (Upstream EvoCUA allows 16.7 MP.)
4. 4.Do not add `--image-min-tokens 1024` just because llama.cpp warns about it, if you pre-resize your screenshots — it will upscale small images and diverge from upstream.

Coordinate convention (app side)

EvoCUA defaults to coordinate_type=relative: the system prompt states "The screen's resolution is 1000x1000" and the model emits a 0..999 grid. Absolute pixels = x * width / 999. Output is a tool_call, not pyautogui (that is the S1 prompt style):

<tool_call>{"name":"computer_use","arguments":{"action":"left_click","coordinate":[x,y]}}</tool_call>

Screenshots must be pre-resized with `smart_resize(factor=32)` by the caller, exactly as the upstream harness does.

How the verification works

harness/ contains everything. Two design decisions make the numbers trustworthy:

1. 1.Test pages are rendered at 1920×1088, which is 34 × 32 — so smart_resize(factor=32) is the identity. Neither transformers nor llama.cpp resizes the image, which removes preprocessing differences from the measurement and leaves only conversion/quantization.
2. 2.Every click target is placed at absolute CSS coordinates, so the ground-truth bbox is a construction, not a hand measurement. make_testpages.py regenerates the whole suite.

Metrics are split deliberately:

Wide targets (e.g. a 1620 px file row) are reported separately: clicking the filename vs the row centre are both correct, so their large deltas say nothing about quantization.

Notes for AMD / ROCm (gfx1201, RDNA4) users

All measurements were made on 2× AMD Radeon AI PRO R9700 (gfx1201 / RDNA4, 32 GB each) under ROCm with a llama.cpp HIP build. gfx1201 numbers are scarce, so:

(Measured with one 1920×1080 screenshot in the prompt = 2067 prompt tokens, 300 generated, median of 3 restarts. 32B numbers are in REPORT.ko.md.)

• —The system ROCm install may not be enough to build. Ours had no hip / hip-lang / hipblas / rocblas CMake packages, so -DGGML_HIP=ON failed at configure. Point ROCM_PATH / CMAKE_PREFIX_PATH at a gfx1201-targeted ROCm SDK that has them (and set LD_LIBRARY_PATH at runtime).
• —⛔ Do not pass `-ngl` when splitting across multiple GPUs. It disables llama.cpp's automatic fitting: common_fit_params: ... n_gpu_layers already set by user ... abort. Manual values failed twice for us (CUBLAS_STATUS_ALLOC_FAILED on hipBLAS handle creation; cudaMalloc failed: out of memory). Passing only -mg <gpu> and leaving -ngl/-ts unset worked first try. llama.cpp knows the real buffer requirements; don't guess layers.
• —⚠️ If VRAM is short, llama.cpp starts fine and silently falls back to host memory (GTT). No error, correct answers, ~9× slower. We measured the same Q6_K at 15.7 tok/s (-c 8192) vs 1.74 tok/s (-c 32768, VRAM saturated). You can only detect this by measuring speed.

Reproducing

# 1) pull the UD tensor allocation out of Unsloth's published GGUF header (~12 MB, not 5 GB)
python harness/extract_ud_recipe.py \
  --repo unsloth/Qwen3-VL-8B-Thinking-GGUF \
  --file Qwen3-VL-8B-Thinking-UD-Q5_K_XL.gguf \
  --verify-against evocua-8b-bf16.gguf --out ud_q5_recipe.txt

# 2) quantize with it. token_embd / output need the dedicated flags, not --tensor-type.
#    Always --dry-run first: it validates the whole recipe in seconds.
llama-quantize --imatrix evocua-8b.imatrix \
  --tensor-type-file ud_q5_recipe.txt \
  --token-embedding-type q5_k --output-tensor-type q6_k \
  evocua-8b-bf16.gguf evocua-8b-UD-Q5_K_XL.gguf Q5_K_M 24

# 3) regenerate the test suite (deterministic: targets are CSS-positioned, not measured)
python harness/make_testpages.py

# 4) score a build against the reference
python harness/coord_test.py --arm gguf --url http://127.0.0.1:8080/v1 --out coord_x.json
python harness/compare_arms.py --baseline results/coord_hf.json --arms "x=coord_x.json"

Credits & licensing

• —Base model: `meituan/EvoCUA-8B-20260105` (Apache-2.0). These files are quantized derivatives of those weights.
• —Unsloth Dynamic (UD) recipes are not ours. The UD-* files here reproduce the per-tensor bit allocation published by Unsloth for the base model (`unsloth/Qwen3-VL-8B-Thinking-GGUF`). Because EvoCUA-8B is a fine-tune of that model, the GGUF tensor names match exactly (399/399), so their allocation transfers. harness/extract_ud_recipe.py reads it out of the GGUF header via an HTTP range request (~12 MB, no full download). Their published imatrix is used as-is where noted. Credit for the recipe belongs to Unsloth; the measurement is ours.
• —llama.cpp commit used: 3173a56471c1753650cd806694145ffd6dcace67 (ggml 0.22.0).
• —Korean full report: `REPORT.ko.md`.