lucataco/gliner2.5-cua-grounder-macos-v2

GLiNER2.5 CUA Grounder · macOS v2

A local, text-only action chooser beneath a computer-use planner, with a separately trained verified-effect gate. Given one planner step and a short menu of fully specified actions, the frozen selector proposes a supplied action ID; the gate estimates whether that proposal will produce the requested application effect, using explicit current-observation evidence.

Fine-tuned from Fastino's GLiNER2.5 Multi. This repository contains self-contained merged FP32 selector weights, loader assets, and the v2.2 verified-effect gate (gate.json + gate_calibration.json). Code, training scripts, and detailed experiment artifacts are maintained separately.

Usage

pip install "gliner2[local]==2.0.0" "torch==2.14.0" "transformers==4.57.6" "huggingface-hub==0.36.2" "sentencepiece>=0.2,<1" "protobuf>=4,<8"

Use AutoExtractor, not the legacy span-only GLiNER2 loader. The example uses GLiNER's public classification scorer and preserves the training task/instruction:

import json
import torch
from huggingface_hub import snapshot_download
from gliner2 import AutoExtractor
from gliner2.classification import Classifier, ClassificationSchema

MODEL_ID = "lucataco/gliner2.5-cua-grounder-macos-v2"
path = snapshot_download(MODEL_ID)  # Set revision="<commit SHA>" to pin all files together.
device = "mps" if torch.backends.mps.is_available() else "cpu"
model = AutoExtractor.from_pretrained(path).float().to(device).eval()
chooser = Classifier(model)

instruction = (
    "Select the supplied action that performs the current planner instruction on the described control. "
    "Match the requested operation, control name, section and value. Choose reobserve when the observation "
    "is incomplete or loading. Choose abstain if no supplied action matches. "
    "Typing directly targets a field and does not require a separate click."
)
candidates = {
    "a1": 'Type "London" into textbox "Departure city"',
    "a2": 'Type "London" into textbox "Destination city"',
    "a3": 'Type "London" into textbox "Passenger name"',
    "reobserve": "Obtain a fresh observation because the current observation is incomplete or loading.",
    "abstain": "Do not act because no supplied action matches the planner instruction.",
}
labels = {f"c{i:02d}": description for i, description in enumerate(candidates.values())}
schema = ClassificationSchema().single("driver_action", labels, instruction=instruction)
text = json.dumps(
    {"planner_step": "Enter London in the Destination city field."},
    ensure_ascii=False, separators=(",", ":"),
)
scores = chooser.score(text, schema)
probabilities = {
    candidate_id: scores.probability("driver_action", label)
    for candidate_id, label in zip(candidates, labels, strict=True)
}
selected_id = max(probabilities, key=probabilities.get)
print(selected_id)  # a2

Keep the model resident across calls. After download, load the local directory with HF_HUB_OFFLINE=1 for offline inference. Root weights already include the adapter; no separate base-model download is needed.

Transformers 4.57.6 may emit a spurious Mistral-regex warning for this non-Mistral tokenizer. Preserve the supplied tokenizer; export and scoring parity were verified.

Gated endpoint (recommended for execution)

The selector alone has a null execution threshold on its own calibration split: no threshold met the 5% error / 20 accepted-case criterion, so raw selector probabilities must not gate execution. Use the bundled gate with explicit current-observation evidence.

In the v2 source tree today (exact training invocation — this is what the gate's selector_signature binds):

uv run gliner-cua cua-gate-choose \
  --model runs/cua-v2-pilot/train/best \
  --gate huggingface/gate.json \
  --calibration huggingface/gate_calibration.json --jsonl

Each line must contain the gate envelope (cua.gate_request_v1) with actual current observation evidence. gliner-cua cua-gate-choose --help documents the envelope fields. gate.execute_probability estimates requested-effect correctness; the nested choice retains the selector's original probabilities. Nonready requests return reobserve with no effect estimate. The gate only covers com.apple.TextEdit and local.browser-forms under the background-save execution contract; other apps return abstain (unmeasured_application). Note: this repo names the gate calibration gate_calibration.json (the training tree calls it calibration.json).

HF-path note: the HF model.safetensors is functionally identical to the training adapter (334/334 state tensors bit-identical; identical logits, proposals, gate features and effect probabilities on spot checks), but the current GatedCUAChooser binds selector_signature to the training path, device (mps), threads (8) and adapter-vs-merged layout, so loading the gate directly from an HF snapshot path raises "Gate was trained with a different target selector/deployment" today. The bundled gate.json / gate_calibration.json verify against the exact invocation above; HF-path gate loading needs a small loader update to accept merged-weight equivalence.

Integration contract

Planner step → application-owned menu → selector proposes ID → gate estimates effect → Driver executes → verify effect

• —Supply one current planner instruction, with complete descriptions of operation, control name, form section, and value. Include both reserved fallback options.
• —Pack compact JSON with planner_step, optional regions (structured, already extracted observations), and recent_outcomes (nonempty history outcome strings). An optional local semantic context is packed as observation.
• —Preserve candidate order and map IDs to c00, c01, etc. Escape the description markers [P], [L], [C], [E], [R], [DESCRIPTION], [EXAMPLE], [OUTPUT] by replacing square brackets with braces; replace parentheses with braces and collapse whitespace. The simple example above needs no escaping.
• —Use the exact driver_action task and instruction above. The selector proposal is the maximum-logit executable candidate; report softmax probabilities at temperature 1. These are not calibrated confidence or TypeSafe's separately defined confidence statistic.
• —The gate never reranks executable candidates or rewrites arguments. It applies two documented hard rules before the learned model: the proposal must carry the goal's resolvable section/window context, and goal/proposal operations (type vs click) must agree when both are determinable. Readiness (explicit complete, non-loading, non-superseded capture within 30 s) is a deterministic precondition; history cannot override it.
• —Reject oversized inputs rather than truncating text or dropping candidates. The integration was tested with a 64 KiB request limit and 4,096 encoded tokens.
• —The application retains immutable tool arguments and capture/element handles, validates returned IDs and freshness, and independently verifies effects.

The tested wire schemas are cua.jev_choice_request_v1 / cua.jev_choice_v1 from CUA PR 3916, pinned at 201732fffd81a40818be7ce2e04269aec962bc42, wrapped for gating as cua.gate_request_v1 / cua.gate_decision_v1. This model generates neither coordinates nor tool calls and does not consume screenshot pixels.

Results

Frozen paired synthetic test (selector, 256 requests)

Inputs and acceptable-ID labels match across models. Executable-case accuracy rose from 77.6% to 99.4%, but abstention accuracy fell from 67.7% to 35.5%. Menus include 8/16/32-choice menus, same-name controls across sections, wrong-operation/value alternatives, equivalent duplicates, missing targets, and partial/loading captures. Workflow vocabularies and goal templates are split; construction rules are shared. These are not real-app execution results.

Fitted selector temperature: 1.8424. Selector execution threshold: None (null means reject all executions — no threshold met the 5% error / 20 accepted-case criterion on the dedicated calibration split). Deploy the gate below for execution decisions.

Verified-effect gate (real held-out traces, frozen selector)

Controlled workflows in real TextEdit (background AX insertion + PID-addressed native save, file-byte verification) and an owned browser form application (isolated profile, DOM-state callbacks), with both apps in every split. Nonready requests (partial/loading/superseded/expired/unknown) are deterministic reobserve decisions. Gate calibration: temperature 1.0, threshold 0.95 (correctness-probability floor 1 - max_error), 120/120 accepted with 0 errors on calibration families. Feature version verified-effect-features-v5-operation-aware; L2 0.001 selected on validation. The two hard context/operation rules never fired on real held-out data (rule_rejected=0 at calibration).

Synthetic benchmark with readiness metadata (design check, not independent evidence)

The diagnostic told the authors the failure modes were "wrong section" and "wrong operation", and two of the fixes are hard rules written for exactly those — so 100% confirms the rules do what they were written to do. The independent evidence is the real held-out table above, which never degraded and on which the rules never fired. See the iteration record in the source tree (docs/V22_GATE_ITERATION.md).

Live gated checks

12/12 passed (6 TextEdit + 6 browser): successful execution, missing-target abstention, "any target" instructions, duplicate valid buttons, nonready refusals, and recovery with a real prior deferral in history. Rejected proposals were not dispatched; successful actions were independently verified.

Local timings are FP32 Apple MPS with eight CPU threads. Warm timing excludes model loading and the first request. These are not hardware-normalized timings or end-to-end workflow speedups. No Jev outputs were used for training or distillation.

Training and export

Menus were resampled, reordered, and rekeyed during training. Labels derive from owned fixture definitions. Live confirmation traces were not used for training. The loss maximizes total probability across all valid candidate IDs. The gate never retrains the transformer.

Export verification (run before upload): all state tensors bit-identical after reloading the merged checkpoint; test-request logits and selections match the selected adapter; gate + calibration hashes bind the frozen selector identity (f6414dcb977688e177c38681c0122586348a224151690d9de4248e6be2f3e2fe).

• —Original adapter SHA-256: 49ec5ea8cc495a5ffb2f8bf99b597cebf8f232ce3f91d5279a617ca485c251c8
• —Merged weights SHA-256: 53d8f9209584270bf8d538cc4bf5bed10fabd8fede150c9c2cb30b303e50e640
• —Gate SHA-256: bb41ea6ba15d6a92253a4ff07bef1541dab91dc872fa210b311e648e2a3ce3e1

Limitations

Intended for experimental grounding of short, well-described menus beneath a planner, plus a controlled TextEdit/browser execution gate. Candidate retrieval, visual perception, long-horizon planning, and tool execution are external. Natural planner language, unfamiliar apps, large or ambiguous menus, adversarial UI text, and unannounced state changes need broader testing. The gate is a controlled target-app pilot (two app IDs, scripted workflows, recurring construction patterns), not a guaranteed error bound on arbitrary applications; actual app outcomes must still be checked, and the gate is not a replacement for Driver's capability/refusal checks. Synthetic cases contain explicit textual cues. Out-of-distribution calibration, multilingual grounding, and the base model's general extraction abilities after this fine-tune have not been evaluated. macOS describes the tested integration; inference itself is ordinary PyTorch.

License and attribution

Fine-tune: lucataco. Base model/library: Fastino AI. Encoder: Microsoft mDeBERTa-v3-base. Apache-2.0, with the encoder's MIT license retained in LICENSE.mdeberta. See NOTICE for modifications. This is an independent experimental release.

@misc{lucataco2026cuagrounderv2,
  title = {GLiNER2.5 CUA Grounder for macOS, v2},
  author = {lucataco},
  year = {2026},
  url = {https://huggingface.co/lucataco/gliner2.5-cua-grounder-macos-v2}
}
@misc{zaratiana2025gliner2efficientmultitaskinformation,
  title = {GLiNER2: An Efficient Multi-Task Information Extraction System with Schema-Driven Interface},
  author = {Urchade Zaratiana and Gil Pasternak and Oliver Boyd and George Hurn-Maloney and Ash Lewis},
  year = {2025},
  eprint = {2507.18546},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.18546}
}