get2rot/WildGUI

WildGUI

This repository hosts a personally reprocessed annotation release for WildGUI, the dataset introduced by Video2GUI.

The original Video2GUI project builds WildGUI from large-scale Internet tutorial videos for GUI agent pretraining. This repository focuses on the open annotation artifacts: the records were regenerated and cleaned following the full annotation workflow, then reformatted to make the data easier to inspect, reuse, and reproduce. It also ships the screenshot frames that the trajectories reference.

What's in this repository

The screenshots for the remaining parts (part16–part19) are hosted in a companion repository: `joker-112/WildGUI_Screenshots`. So the annotations for every part live here, while their screenshot frames are split between this repo (part1–part15) and the companion repo (part16–part19).

Annotation File Layout

The processed annotation release is organized as JSONL shards, one per part:

wildgui_part1.jsonl
wildgui_part2.jsonl
...
wildgui_part19.jsonl

Each line is one task-level JSON object. The original multi-task annotations entries were split by task_id, so each record contains a single GUI task and its ordered action trajectory.

Record Structure

Top-level fields:

Each trajectories item is an action object. Common action fields include:

Example

{
  "video_id": "...",
  "segment_index": 0,
  "task_id": 0,
  "instruction": "Use a component library to find a design element and apply it to an AI app builder.",
  "dense_caption": "...",
  "plan": "step1: ...",
  "platform": "web",
  "software": "Google Chrome",
  "website": "example.com",
  "trajectories": [
    {
      "timestamp": "00:18",
      "action_type": "moveTo",
      "grounding_instruction": "Move the mouse cursor over the target component.",
      "action_reason": "...",
      "core_change_reason": "...",
      "core_change": "...",
      "use_grounding": false
    }
  ]
}

Screenshots (part1–15)

Screenshot frames are grouped by the same partN shards and packed into uncompressed tar archives (the frames are already JPEG-compressed):

screenshots/
  part1/
    wildgui_part1_images_000001.tar
    wildgui_part1_images_000002.tar
    ...
  part2/
    ...
  ...
  part15/
    ...

Inside each tar, every frame is stored under a per-video directory:

{video_id}/screenshot_{MM_SS}.jpg

{MM_SS} is the action timestamp normalized to zero-padded minutes_seconds (e.g. the annotation timestamp "00:18" → screenshot_00_18.jpg).

Linking an annotation to its screenshot

Each trajectory action maps to exactly one frame. Given a record's video_id and an action's timestamp:

1. 1.Take the partN matching the annotation shard (e.g. wildgui_part3.jsonl → screenshots/part3/). For part16–part19, fetch the frames from `joker-112/WildGUI_Screenshots` instead.
2. 2.Normalize the timestamp to MM_SS: timestamps like "00:18", "1:05", or a raw second count are converted to total minutes_seconds, each part zero-padded to two digits.
3. 3.The frame is {video_id}/screenshot_{MM_SS}.jpg, found inside one of that part's wildgui_part{N}_images_*.tar shards.

def timestamp_to_suffix(ts: str) -> str:
    """'00:18' -> '00_18', '1:05' -> '01_05', '78' -> '01_18'."""
    ts = str(ts).strip()
    if ":" in ts:
        total = 0
        for part in ts.split(":"):
            total = total * 60 + int(part)
    else:
        total = int(float(ts))
    minutes, seconds = divmod(total, 60)
    return f"{minutes:02d}_{seconds:02d}"

A small fraction of annotated frames may be missing from the packed shards (source frame unavailable at pack time); treat a missing {video_id}/screenshot_{MM_SS}.jpg as a skippable example rather than an error.

Loading

Annotations:

from datasets import load_dataset

dataset = load_dataset(
    "xwm/WildGUI",
    data_files="wildgui_part*.jsonl",
    split="train",
)

For local files:

from datasets import load_dataset

dataset = load_dataset(
    "json",
    data_files="/path/to/wildgui_part*.jsonl",
    split="train",
)

Screenshots (download and unpack one part):

hf download xwm/WildGUI \
  --repo-type dataset \
  --include "screenshots/part1/*" \
  --local-dir ./wildgui

for t in ./wildgui/screenshots/part1/*.tar; do
  tar -xf "$t" -C ./wildgui_frames
done

Intended Use

WildGUI is intended for research on GUI agents, GUI grounding, action prediction, interaction trajectory modeling, and multimodal agent pretraining.

The annotations are automatically derived from tutorial videos and may contain noise. Users should validate the data for their own downstream training or evaluation settings, especially when relying on spatial grounding. For actions with use_grounding: false, use the natural-language action context but avoid treating removed coordinates or parameters as valid supervision.

Citation

If you use this reprocessed annotation release, please cite the Video2GUI paper:

@misc{xiong2026video2gui,
  title = {Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining},
  author = {Xiong, Weimin and Gu, Shuhao and Ye, Bowen and Yue, Zihao and Li, Lei and Song, Feifan and Li, Sujian and Tian, Hao},
  year = {2026},
  eprint = {2605.14747},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL},
  doi = {10.48550/arXiv.2605.14747},
  url = {https://arxiv.org/abs/2605.14747}
}