get2rot/WildGUI

1---2pretty_name: WildGUI3license: cc-by-nc-4.04language:5- en6tags:7- gui-agents8- gui-grounding9- interaction-trajectories10- video2gui11- wildgui12- web13- desktop14- mobile15size_categories:16- 10M<n<100M17---18 19# WildGUI20 21This repository hosts a personally reprocessed annotation release for **WildGUI**, the dataset introduced by Video2GUI.22 23The original Video2GUI project builds WildGUI from large-scale Internet tutorial videos for GUI agent pretraining. This repository focuses on the open annotation artifacts: the records were regenerated and cleaned following the full annotation workflow, then reformatted to make the data easier to inspect, reuse, and reproduce. It also ships the screenshot frames that the trajectories reference.24 25## What's in this repository26 27| Content | Coverage | Location |28|---|---|---|29| **Annotations** (JSONL) | `part1`–`part19` (all parts) | `wildgui_part{N}.jsonl` |30| **Screenshots** (tar shards) | `part1`–`part15` | `screenshots/part{N}/` |31 32The screenshots for the remaining parts (`part16`–`part19`) are hosted in a33companion repository:34[`joker-112/WildGUI_Screenshots`](https://huggingface.co/datasets/joker-112/WildGUI_Screenshots).35So the annotations for every part live here, while their screenshot frames are36split between this repo (`part1`–`part15`) and the companion repo37(`part16`–`part19`).38 39## Annotation File Layout40 41The processed annotation release is organized as JSONL shards, one per part:42 43```text44wildgui_part1.jsonl45wildgui_part2.jsonl46...47wildgui_part19.jsonl48```49 50Each line is one task-level JSON object. The original multi-task `annotations` entries were split by `task_id`, so each record contains a single GUI task and its ordered action trajectory.51 52## Record Structure53 54Top-level fields:55 56| Field | Type | Description |57|---|---:|---|58| `video_id` | string | Source video identifier. |59| `segment_index` | integer | Segment index within the source video. |60| `task_id` | integer | Task identifier after splitting the original annotations. |61| `instruction` | string | Natural-language task instruction. |62| `dense_caption` | string | Dense natural-language summary of the GUI behavior. |63| `plan` | string | High-level task plan. |64| `platform` | string | Platform context, such as web, desktop, or mobile. |65| `software` | string | Application or software used in the interaction. |66| `website` | string | Website or web service, when applicable. |67| `trajectories` | list | Ordered GUI action sequence for the task. |68 69Each `trajectories` item is an action object. Common action fields include:70 71| Field | Type | Description |72|---|---:|---|73| `timestamp` | string | Action timestamp in the source video segment. |74| `action_type` | string | Action category, such as `moveTo`, click, typing, scrolling, or keyboard operation. |75| `grounding_instruction` | string | Natural-language description of the target GUI element or action target. |76| `action_reason` | string | Reason for performing the action. |77| `action_parameters` | object, optional | Grounded parameters such as `point`, `bbox`, `text`, `key`, `start_point`, `end_point`, `direction`, or duration fields. This field is omitted when reliable grounding is unavailable. |78| `core_change_reason` | string | Explanation of the expected or observed GUI state change. |79| `core_change` | string | Description of the main GUI state change after the action. |80| `effects_on_success` | string, optional | Additional note on how the action affects task completion. |81| `finish_reason` | string, optional | Reason the trajectory or task is considered finished. |82| `use_grounding` | boolean | Whether the grounded action parameters should be used. |83 84## Example85 86```json87{88  "video_id": "...",89  "segment_index": 0,90  "task_id": 0,91  "instruction": "Use a component library to find a design element and apply it to an AI app builder.",92  "dense_caption": "...",93  "plan": "step1: ...",94  "platform": "web",95  "software": "Google Chrome",96  "website": "example.com",97  "trajectories": [98    {99      "timestamp": "00:18",100      "action_type": "moveTo",101      "grounding_instruction": "Move the mouse cursor over the target component.",102      "action_reason": "...",103      "core_change_reason": "...",104      "core_change": "...",105      "use_grounding": false106    }107  ]108}109```110 111## Screenshots (part1–15)112 113Screenshot frames are grouped by the same `partN` shards and packed into114uncompressed tar archives (the frames are already JPEG-compressed):115 116```text117screenshots/118  part1/119    wildgui_part1_images_000001.tar120    wildgui_part1_images_000002.tar121    ...122  part2/123    ...124  ...125  part15/126    ...127```128 129Inside each tar, every frame is stored under a per-video directory:130 131```text132{video_id}/screenshot_{MM_SS}.jpg133```134 135`{MM_SS}` is the action timestamp normalized to zero-padded `minutes_seconds`136(e.g. the annotation timestamp `"00:18"` → `screenshot_00_18.jpg`).137 138### Linking an annotation to its screenshot139 140Each trajectory action maps to exactly one frame. Given a record's `video_id`141and an action's `timestamp`:142 1431. Take the `partN` matching the annotation shard (e.g. `wildgui_part3.jsonl` →144   `screenshots/part3/`). For `part16`–`part19`, fetch the frames from145   [`joker-112/WildGUI_Screenshots`](https://huggingface.co/datasets/joker-112/WildGUI_Screenshots) instead.1462. Normalize the timestamp to `MM_SS`: timestamps like `"00:18"`, `"1:05"`, or a147   raw second count are converted to total `minutes_seconds`, each part148   zero-padded to two digits.1493. The frame is `{video_id}/screenshot_{MM_SS}.jpg`, found inside one of that150   part's `wildgui_part{N}_images_*.tar` shards.151 152```python153def timestamp_to_suffix(ts: str) -> str:154    """'00:18' -> '00_18', '1:05' -> '01_05', '78' -> '01_18'."""155    ts = str(ts).strip()156    if ":" in ts:157        total = 0158        for part in ts.split(":"):159            total = total * 60 + int(part)160    else:161        total = int(float(ts))162    minutes, seconds = divmod(total, 60)163    return f"{minutes:02d}_{seconds:02d}"164```165 166A small fraction of annotated frames may be missing from the packed shards167(source frame unavailable at pack time); treat a missing168`{video_id}/screenshot_{MM_SS}.jpg` as a skippable example rather than an error.169 170## Loading171 172Annotations:173 174```python175from datasets import load_dataset176 177dataset = load_dataset(178    "xwm/WildGUI",179    data_files="wildgui_part*.jsonl",180    split="train",181)182```183 184For local files:185 186```python187from datasets import load_dataset188 189dataset = load_dataset(190    "json",191    data_files="/path/to/wildgui_part*.jsonl",192    split="train",193)194```195 196Screenshots (download and unpack one part):197 198```bash199hf download xwm/WildGUI \200  --repo-type dataset \201  --include "screenshots/part1/*" \202  --local-dir ./wildgui203 204for t in ./wildgui/screenshots/part1/*.tar; do205  tar -xf "$t" -C ./wildgui_frames206done207```208 209## Intended Use210 211WildGUI is intended for research on GUI agents, GUI grounding, action prediction, interaction trajectory modeling, and multimodal agent pretraining.212 213The annotations are automatically derived from tutorial videos and may contain noise. Users should validate the data for their own downstream training or evaluation settings, especially when relying on spatial grounding. For actions with `use_grounding: false`, use the natural-language action context but avoid treating removed coordinates or parameters as valid supervision.214 215## Citation216 217If you use this reprocessed annotation release, please cite the Video2GUI paper:218 219```bibtex220@misc{xiong2026video2gui,221  title = {Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining},222  author = {Xiong, Weimin and Gu, Shuhao and Ye, Bowen and Yue, Zihao and Li, Lei and Song, Feifan and Li, Sujian and Tian, Hao},223  year = {2026},224  eprint = {2605.14747},225  archivePrefix = {arXiv},226  primaryClass = {cs.CL},227  doi = {10.48550/arXiv.2605.14747},228  url = {https://arxiv.org/abs/2605.14747}229}230```231