GumbiiDigital/macos-gui-agent-qwen2.5-vl-3b

macos-gui-agent-qwen2.5-vl-3b

LoRA fine-tune of Qwen/Qwen2.5-VL-3B-Instruct for macOS GUI automation from screenshots.

This model is intended to look at a macOS/app screenshot plus a natural-language task and emit the next GUI action as a JSON object. It is an experimental screenshot-to-action adapter, not a complete desktop automation product.

Repository:

• —Model: https://huggingface.co/GumbiiDigital/macos-gui-agent-qwen2.5-vl-3b
• —Training dataset: https://huggingface.co/datasets/GumbiiDigital/macos-gui-agent-train
• —Base model: https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct
• —Trackio run: https://GumbiiDigital-mlintern-macgui.hf.space?project=macos-gui-agent&runs=macos-gui-agent-v2&sidebar=collapsed

What this is for

Given:

1. 1.A screenshot.
2. 2.A task such as click the UI element Create new... or Open the Settings tab.

The adapter should respond with JSON similar to:

{
  "action": "click",
  "position": [0.1575, 0.134],
  "value": null
}

position is expected to use normalized screen coordinates in [x, y], where both values should be between 0 and 1.

The target action schema used during training was:

{
  "action": "click|type|scroll|drag|hotkey",
  "position": [0.0, 0.0],
  "value": "text to type, if applicable"
}

Some training examples also include metadata keys such as app, category, or target_description.

Important: this is a LoRA adapter

This repository contains a PEFT/LoRA adapter, not a merged full model. Load it on top of the base model Qwen/Qwen2.5-VL-3B-Instruct.

Main adapter file:

• —adapter_model.safetensors — 594,546,704 bytes

LoRA config:

• —rank r: 64
• —alpha: 16
• —dropout: 0.05
• —target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• —PEFT version: 0.18.1

Quick start

Install dependencies:

pip install torch transformers peft qwen-vl-utils pillow accelerate

Run inference:

import torch
from PIL import Image
from peft import PeftModel
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration

BASE_MODEL = "Qwen/Qwen2.5-VL-3B-Instruct"
ADAPTER = "GumbiiDigital/macos-gui-agent-qwen2.5-vl-3b"

processor = AutoProcessor.from_pretrained(
    BASE_MODEL,
    min_pixels=256 * 28 * 28,
    max_pixels=1344 * 28 * 28,
)

base_model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    BASE_MODEL,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, ADAPTER)
model.eval()

image = Image.open("screenshot.png").convert("RGB")

task = "click the UI element Create new..."
messages = [
    {
        "role": "system",
        "content": [
            {
                "type": "text",
                "text": (
                    "You are a macOS automation assistant that controls the computer "
                    "via the Accessibility API. Given a screenshot and a task, output "
                    "the next action as a JSON object with keys: 'action' "
                    "(click|type|scroll|drag|hotkey), 'position' [x_norm, y_norm] "
                    "normalized 0-1, and 'value' for text if applicable."
                ),
            }
        ],
    },
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": f"Task: {task}"},
        ],
    },
]

text = processor.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

inputs = processor(
    text=[text],
    images=[image],
    padding=True,
    return_tensors="pt",
).to(model.device)

with torch.no_grad():
    generated_ids = model.generate(
        **inputs,
        max_new_tokens=96,
        do_sample=False,
    )

trimmed = [
    output_ids[len(input_ids):]
    for input_ids, output_ids in zip(inputs.input_ids, generated_ids)
]

output = processor.batch_decode(
    trimmed,
    skip_special_tokens=True,
    clean_up_tokenization_spaces=False,
)[0]
print(output)

Expected style:

{"action": "click", "position": [0.1575, 0.134], "value": null}

Training data

Training dataset: GumbiiDigital/macos-gui-agent-train

The dataset contains 6,000 screenshot/action examples built from:

• —2,000 examples from ritzzai/GUI-R1
• —2,000 examples from macpaw-research/GUIrilla-Task
• —2,000 examples from macpaw-research/Screen2AX-Task

Dataset shape:

{
    "messages": [
        {"role": "system", "content": [{"type": "text", "text": "..."}]},
        {"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "Task: ..."}]},
        {"role": "assistant", "content": [{"type": "text", "text": "{...action json...}"}]},
    ],
    "images": [PIL image]
}

Dataset audit at training time:

• —total rows: 6,000
• —action distribution:
• —click: 5,934 examples
• —type: 66 examples
• —non-null value fields: 0
• —target positions normalized to 0-1: yes, 5,934/5,934 examples with positions were in range
• —metadata keys:
• —app: 2,000 examples
• —category: 2,000 examples
• —target_description: 2,000 examples

This means the adapter is primarily a click-location model. It should not be considered trained for reliable typing, scrolling, dragging, or hotkeys.

Training procedure

Training command used TRL SFTTrainer with PEFT LoRA.

Core settings:

• —base model: Qwen/Qwen2.5-VL-3B-Instruct
• —epochs: 1
• —dataset rows: 6,000
• —optimizer steps: 750
• —per-device train batch size: 1
• —gradient accumulation: 8
• —learning rate: 2e-5
• —scheduler: cosine
• —warmup ratio: 0.03
• —max length: 2048
• —precision: bf16
• —gradient checkpointing: enabled
• —save strategy: epoch

Framework versions:

• —TRL: 1.3.0
• —Transformers: 4.57.6
• —PyTorch: 2.11.0+cu130
• —Datasets: 4.8.4
• —PEFT: 0.18.1
• —Tokenizers: 0.22.2

Final training state:

• —global step: 750
• —epoch: 1.0
• —final loss: 7.015
• —final mean token accuracy: 0.0914150146767497
• —final tokens seen: 8,578,259
• —adapter size: 594,546,704 bytes

Training loss summary from trainer_state.json:

• —first logged loss: 16.9928
• —final logged loss: 7.015
• —minimum logged loss: 6.9813
• —mean first 10 logged losses: 13.01959
• —mean last 10 logged losses: 7.01031
• —mean first 10 token accuracy: 0.04403
• —mean last 10 token accuracy: 0.09214

Evaluation status

No proper held-out benchmark has been run yet. The numbers below are smoke-test checks, not a real validation score.

A 6-example in-training smoke sample gave:

• —JSON-valid outputs: 6/6
• —action label match: 6/6
• —median normalized position L2 error: 0.1147

The same 6 examples using the unfine-tuned base model gave:

• —JSON-valid outputs: 6/6
• —action label match: 1/6
• —outputs often used left_click, raw pixel coordinates, or extra markdown fences

Observed adapter behavior from smoke testing:

• —Strong improvement in following the requested JSON style compared with the base model.
• —Strong improvement in using click instead of non-schema actions like left_click.
• —Usually emits normalized coordinates, but at least one smoke output still produced raw pixel coordinates ([763, 842]).
• —Coordinate accuracy is inconsistent. Some examples are close; others are far from target.

Practical verdict:

• —Good enough as a first experimental adapter proving the pipeline works.
• —Not yet good enough to trust for autonomous macOS control without guardrails.
• —Needs a held-out evaluation set and better action diversity before being called production-ready.

Limitations

Major limitations:

1. 1.The dataset is almost entirely click actions.
2. 2.5,934/6,000 examples are click.
3. 3.Only 66 examples are type.
4. 4.No meaningful training coverage for scroll, drag, hotkey, or text entry values.

1. 1.The model may still output invalid coordinate scale.
2. 2.Target format is normalized 0-1.
3. 3.Smoke testing found at least one adapter output using raw pixel coordinates.

1. 1.No held-out validation score yet.
2. 2.The current evaluation is an in-training smoke test.
3. 3.It proves formatting improved, not real generalization.

1. 1.The model does not execute actions.
2. 2.It only predicts JSON.
3. 3.A separate macOS Accessibility API executor would be required.
4. 4.Any executor should validate JSON, clamp coordinates, require safety checks, and avoid destructive actions.

1. 1.It is not a general web/desktop agent.
2. 2.It was trained on GUI screenshots and action labels.
3. 3.It does not have planning, memory, browsing, or tool execution built in.

Recommended safety wrapper

Before sending actions to a real computer, wrap outputs with checks:

• —parse strict JSON only
• —reject markdown/fenced code
• —require action in allowed set
• —clamp position to [0, 1]
• —reject raw pixel coordinates unless explicitly converted
• —require confirmation for destructive actions
• —add a screenshot-before/screenshot-after loop
• —keep a human abort hotkey

Example validation rule:

def validate_action(action):
    allowed = {"click", "type", "scroll", "drag", "hotkey"}
    if action.get("action") not in allowed:
        raise ValueError("unsupported action")
    pos = action.get("position")
    if action.get("action") in {"click", "drag"}:
        if not isinstance(pos, list) or len(pos) != 2:
            raise ValueError("missing normalized position")
        x, y = pos
        if not (0 <= x <= 1 and 0 <= y <= 1):
            raise ValueError("position must be normalized 0-1")
    return action

Intended users

This adapter is intended for researchers and builders experimenting with:

• —GUI grounding
• —macOS computer-use agents
• —screenshot-to-action models
• —Accessibility API automation
• —VLM fine-tuning pipelines

It is not intended for unsupervised control of a personal computer.

How to improve next

Recommended next steps:

1. 1.Build a held-out test split and report exact metrics.
2. 2.Add action-balanced data for type, scroll, drag, and hotkey.
3. 3.Include non-null value examples for text entry.
4. 4.Enforce normalized-coordinate output during decoding or post-processing.
5. 5.Evaluate coordinate error by dataset source and task category.
6. 6.Add real macOS screenshot rollouts with a human approval gate.
7. 7.Consider a second training pass after fixing the output schema and balancing examples.

Citation

@software{vonwerra2020trl,
  title   = {{TRL: Transformers Reinforcement Learning}},
  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
  license = {Apache-2.0},
  url     = {https://github.com/huggingface/trl},
  year    = {2020}
}