hugging-apps/scalecua-gui-agent

ScaleCUA — GUI Action Prediction Demo

ScaleCUA scales computer-use agents (CUAs) with verifiable task synthesis and efficient online RL. This demo loads the ScaleCUA-Qwen3-VL-8B OSWorld checkpoint (extreme1228/ScaleCUA-qwen3-vl-osworld) and performs single-step action prediction: given a desktop screenshot and a task instruction, the model predicts the next GUI action (click, type, scroll, hotkey, etc.) with pixel coordinates.

The model is a Qwen3VLForConditionalGeneration fine-tuned on the SYS_PROMPT_IN_SCREENSHOT_OUT_ACTION prompt (screenshot-only observation, computer_13 action space) from the SCALE-CUA repository. Coordinates are raw pixels in a 1920×1080 screen space.

Note: This is a single-step predictor, not a full closed-loop agent. The full ScaleCUA agent runs inside OSWorld/ScienceBoard Docker sandboxes, which is out of scope for a simple demo. Upload your own screenshot, type a task, and see what the model would do next.