CoolFace
Apppublic

hugging-apps/ui-mate-27b-demo

sourceHugging Faceupdated 1mo agoView on Hugging Face
2likes
App README

UI-Mate-27B — next GUI action from a screenshot

A single-step demo of tencent/UI-Mate-27B, Tencent HY Frontier's open-weight foundation computer-use agent (27B, built on Qwen3.6-27B, Apache-2.0).

Give it a desktop screenshot and a task instruction. The demo runs exactly one step of the reference agent loop: UI-Mate reasons over the visible state in a <think> block, summarises the move in an <action> block, and emits a structured computer_use tool call. That call is translated into runnable pyautogui code and the target coordinate is plotted back onto the screenshot.

Nothing is executed — the model reasons in a normalized 1000×1000 screen space and the parser rescales coordinates to the screenshot's own pixel size, but actually driving a desktop needs the external UI-Mate harness.

Fidelity

ui_mate_agent.py is Tencent/UI-Mate's agents/ui_mate_agent.py, vendored verbatim (Apache-2.0). The system prompt, the tool schema, the smart_resize screenshot preprocessing, the XML tool-call parser and the pyautogui translation are therefore the authors' own. The marker plotting is adapted from that repo's examples/run_agent.py.

The only substitution is transport: the reference agent talks to an OpenAI-compatible vLLM endpoint, while this Space generates locally with transformers (Qwen3_5ForConditionalGeneration, bf16) on ZeroGPU. Sampling defaults match the reference configuration (temperature=1.0, top_p=0.95, thinking enabled).

Examples

The bundled screenshots are the reference repo's own single-step examples: step-0 frames from OSWorld tasks captured on its Ubuntu 22.04 image at 1920×1080, each paired with that task's own instruction. Both projects are Apache-2.0.

Notes

  • —27B in bf16 is ~55 GB of weights, so inference runs on a full ZeroGPU card (size="xlarge"), which consumes ZeroGPU quota at 2× the usual rate.
  • —The default generation budget (768 tokens) covers a typical reasoning + action + tool-call reply. Long-horizon tasks may want more; raise it under Advanced options.
  • —Multi-step context is supported through the Previous actions box, which fills the agent's Previous actions: history field.