zhanwenchen/personalgui-accel-data

GUI-agent evaluation: code, per-episode records, and captured media

This repository holds everything behind a study of whether a language model can operate real software when its only input is a video feed of the screen and its only output is a keyboard and mouse. The rendered report lives in a separate Space; this is the material it is computed from.

What the setup is

Two machines, joined by two cables and no software link. One runs the agent. The other runs the task and is completely unmodified — nothing is installed on it and it exposes no interface to the first machine. A capture card digitises its screen and feeds those frames to the agent machine. A USB device that presents itself as an ordinary keyboard and mouse carries actions back. The agent cannot read application state, click by element id, or call a function; it sees pixels and sends input, which is the same access a person has.

The hardware input device was chosen to avoid writing a virtual-HID driver. It is used only to send input to the researcher's own machine.

The three tasks

• —A — vehicle intercept. The agent commands a tank in a driving game and must find and destroy one enemy vehicle before it escapes. Runs live, because the agent's own actions change what happens next. The game adjudicates the outcome.
• —B — sponsor skip. Press the skip key while a recorded video is inside a paid advertisement read.
• —C — chart pattern. Click the button naming a technical pattern at the moment it completes on a replayed price chart.

Tasks B and C replay recorded frames, so every model sees byte-identical input.

Layout

Frames and videos are stored with git-lfs.

Reading the results

Two files are worth opening before any table. FINDINGS.md records what was learned and what was decided; RETRACTIONS.md lists what was published and later found to be wrong. Several headline numbers in earlier report versions were withdrawn — a precision metric that charged presses it should have credited, a speed comparison that measured two different prompts, and an episode boundary that discarded five seconds of real gameplay. The retraction list is part of the result, not an appendix to it.

Numbers in the report are regenerated from the records on every build rather than transcribed, after three separate incidents of a figure drifting from the data it described.

A note on the captured frames

The recorded frames and videos are of a personal desktop. They incidentally contain a Windows username, local file paths, browser bookmarks, and a game account name. None of it is the subject of the research; it is the background of a screen that happened to be recorded. Please treat it accordingly.

Licence

CC BY-NC 4.0. The captured game footage remains the property of its publisher.