hugging-apps/hybrid-routing-agent

Hybrid GUI-MCP Computer-Use Agent

Interactive demo of the policy released with "Screenshots or Tools? Eliciting Tool Use and Managing Multimodal Context in Hybrid GUI-MCP Computer-Use Agents" (paper · code).

Give the agent a desktop screenshot and a task instruction. It predicts its next single step and routes it one of two ways:

• —GUI action — a computer_use call (left_click, type, left_click_drag, scroll, terminate, …) with coordinates on the model's 1000×1000 grid, rendered as a marker on the screenshot and as the pyautogui code OSWorld would execute.
• —MCP tool call — a high-level tool from the OSWorld-MCP registry, e.g. libreoffice_calc.transpose_range(source_range="B2:F5", target_cell="B8"), rendered as MCP:<tool>:<json>.

Uncheck Expose MCP tools in Advanced settings to reproduce the paper's GUI-only baseline and watch the same screenshot route to a click instead of a tool call.

Model

Because it is a Thinking model, every response starts with a <think> trace; it is streamed live into the Raw model output panel.

Fidelity to the reference implementation

The prompt, the tool retrieval and the response parser are ported from the authors' code:

• —system / user prompt — OSWorld-main/mm_agents/hybrid_agent_local.py::_build_messages_owl
• —MCP tool retrieval — agents/tool_retriever.py (BM25 over tools/tools_registry.json, 120 tools across 9 app namespaces), vendored verbatim
• —tool-call parsing and pyautogui lowering — OSWorld-main/mm_agents/owl_parser.py (input_swap=False, as the agent uses)
• —decoding — greedy (temperature=0), repetition_penalty=1.05, stop at </tool_call>

Two deliberate deviations, both forced by the demo setting:

1. 1.OSWorld always runs at 1920×1080, so the reference maps the model's 1000×1000 coordinate grid onto that fixed screen. Here it is mapped onto whatever screenshot you upload.
2. 2.Uploads are down-sampled to at most 1920×1088 pixels (the training resolution) to keep the image-token count and latency in the regime the policy was trained on.

This is a single-step demo: there is no live desktop, so nothing is executed and there is no multi-step rollout. You can approximate history with the optional Previous actions field.

Examples

The three examples are the authors' own case-study screenshots for OSWorld task eb03d19a ("Apply matrix transposition to the table in B2:F5 and paste the transposed table at B8"), taken from results/cases/calc_transpose/ in redai-infra/hybrid-routing-agent (Apache-2.0): the initial state, the state after a pure-GUI paste goes wrong, and the state after the transposed table has landed.

License

Apache-2.0, following the source repository. Model weights are governed by their own license.