gui-wm/spine
GUI World Model — Spine Transitions (s, a, s') transitions collected by walking task instructions on a live Ubuntu desktop. Every state is captured from the running machine: a screenshot, the accessibility tree as XML, and the rendered element table the model reads. This set is spine only — the path an agent actually took. No branches. Where the instructions come from instruction_source what it is agentnet Human recordings of people using their own… See the full description on the dataset page: https://huggingface.co/datasets/gui-wm/spine.
GUI World Model — Spine Transitions
(s, a, s') transitions collected by walking task instructions on a live Ubuntu desktop. Every state is captured from the running machine: a screenshot, the accessibility tree as XML, and the rendered element table the model reads.
This set is spine only — the path an agent actually took. No branches.
Where the instructions come from
The agent is OpenCUA-7B throughout. The desktop is OSWorld's Ubuntu VM at 1920×1080.
Columns that say what a row is
Our internal pipeline names (mode_a, mode_b) are deliberately absent: they name which of our programs ran, not what produced the data. These do:
State columns: before_screenshot · after_screenshot (PNG bytes), before_a11y_xml · after_a11y_xml, before_a11y_txt · after_a11y_txt, before_awb_txt · after_awb_txt (the AgentWorldBench format-A table: tag · name · text · class · description · position (top-left x&y) · size (w&h)).
What was removed, and what was only labelled
Removed — one `(s, a)` reaching two different `s'`. Measured on a 700-trajectory sample: 111 such groups, and they are not measurement noise. Opening Thunderbird sometimes shows a welcome page and sometimes does not, so the same click lands in genuinely different states. Keeping one arm would teach a determinism the desktop does not have, and nothing in the record says which arm was the accident, so both go.
Labelled, not removed — repeated `(s, a, s')`. A repeat is usually a shared opening: two tasks bootstrapped into the same application begin on the same screen. Deleting the later copies would punch holes in the trajectories that produced them, and each AWB example is built from its whole root-to-edge chain — a fifteen-step walk missing its first four would publish its fifth step as a one-turn example. So the row stays and carries duplicate_of. Filter on it when training on individual transitions; ignore it when building histories.
Kept on purpose — transitions where the table does not change. About a quarter of them. An action that changes nothing is a fact about the interface, and a model that never sees one learns that every action changes something.
Restoring the folder layout
The parquet is the delivery form, not a lossy one. guiwm_collector.dataset.pack.materialize() writes the eight-files-per-step directory tree back out, byte for byte — verified on a full trajectory: 176 files, 176 identical.