Aoraku/VG-GUI-Bench

VG-GUI-Bench

Video-Guided GUI Agent Benchmark — the benchmark contribution of our ECCV 2026 paper [Bridging VideoQA and Video-Guided Agentic Tasks via Generalized Keyframe Extraction](https://arxiv.org/abs/2606.29445).

  

Authors. Sunqi Fan, Qingle Liu, Runqi Yin, Meng-Hao Guo, Shuojin Yang (Tsinghua University).

What is VG-GUI-Bench?

Recent Multimodal Large Language Models (MLLMs) achieve strong results on Video Question Answering (VideoQA), but existing benchmarks mostly probe shallow visual perception and rarely test whether a model can learn a procedure from a tutorial video and carry it out as a long-horizon interactive task. VG-GUI-Bench closes this gap.

Given a YouTube tutorial video that demonstrates how to accomplish a task on a mobile device (e.g. "How To Change Discord Password"), a GUI agent must:

1. 1.Understand the workflow from reference frames drawn from the tutorial video (scene keyframes, uniform samples, annotated frames, or algorithmically searched keyframes);
2. 2.Localize the current progress from its previous action history and the current screen; and
3. 3.Predict the exact next action — one of CLICK, SCROLL, TYPE, PRESS, ZOOM, FINISH — on the current screen.

This makes VG-GUI-Bench a testbed for video in-context learning: transferring procedural knowledge from an instructional video to grounded, step-by-step decision making.

This dataset is a processed and re-annotated version built on top of the [MONDAY](https://huggingface.co/datasets/runamu/MONDAY) dataset, curated and packaged for the video-guided GUI agent evaluation described in the paper.

Dataset Statistics

Action type distribution

Repository Layout

VG-GUI-Bench/
├── ours_data.json                 # Step-level action annotations (main label file)
├── ytb_video/                     # Source tutorial videos, one .mp4 per episode (keyed by ep_id)
│   ├── 07hF8RAFgIc.mp4
│   └── ...
└── images/                        # Pre-rendered frames, per reference mode → per episode → frames
    ├── origin/                    #   Scene-timestamp keyframes (cropped to phone screen)
    ├── origin_no_cut/             #   Scene-timestamp keyframes (full frame)
    ├── annotation/                #   Frames with a red bounding box on the target (cropped)
    ├── annotation_no_cut/         #   Frames with a red bounding box on the target (full frame)
    ├── uniform_5/                 #   5 uniformly sampled frames (cropped)
    ├── uniform_5_no_cut/          #   5 uniformly sampled frames (full frame)
    ├── uniform_10/                #   10 uniformly sampled frames (cropped)
    └── uniform_10_no_cut/         #   10 uniformly sampled frames (full frame)

Inside every images/<mode>/ directory the frames are grouped by episode id, e.g. images/origin/07hF8RAFgIc/frame_0000.png, frame_0001.png, …

Data Schema

ours_data.json is a single JSON object with one key, "ours", whose value is a list of 100 episodes. Each episode is a list of steps, and each step has the following fields:

Each `Action` object:

All coordinates are normalized to `[0, 1]` relative to the screen width/height.

Minimal example

{
  "ep_id": "SIjOxM9jVj8",
  "goal": "How To Change Discord Password 2021 | Discord Mobile App",
  "img_filename": "SIjOxM9jVj8/frame_0000",
  "action_list": [
    {
      "action_type_id": 4,
      "action_type_text": "click",
      "annot_position": [0.862, 0.079, 0.062, 0.116],
      "touch": [0.137, 0.893],
      "lift":  [0.137, 0.893],
      "type_text": ""
    }
  ]
}

Reference-Frame Modes

The benchmark studies how different visual-context strategies affect agent performance. The pre-rendered images/ directories cover 8 of these modes (origin, annotation, uniform_5, uniform_10, each in cut / no_cut variants). The evaluation code additionally supports algorithmic keyframe-search modes (tasker, bfs, gbfs, dijkstra) and video-agent modes (videoagent, videotree), which are produced on demand from the source videos — see the code repository.

Usage

import json
from huggingface_hub import snapshot_download

local_dir = snapshot_download(repo_id="Aoraku/VG-GUI-Bench", repo_type="dataset")

data = json.load(open(f"{local_dir}/ours_data.json"))["ours"]
print(len(data), "episodes")
for step in data[0]:                     # iterate over one episode
    print(step["img_filename"], step["action_list"][0]["action_type_text"])

For the full data-processing pipeline, reference-mode generation, and the four evaluation metrics (Accuracy, Completion, Efficiency, PIR), see the official code: https://github.com/VG-GUI-TASKER/VG-GUI-TASKER (VG-GUI-Bench/).

License

Released under the MIT License. The underlying videos remain the property of their respective YouTube uploaders and are provided for research use only.

Citation

@inproceedings{fan2026bridging,
  title     = {Bridging VideoQA and Video-Guided Agentic Tasks via Generalized Keyframe Extraction},
  author    = {Fan, Sunqi and Liu, Qingle and Yin, Runqi and Guo, Meng-Hao and Yang, Shuojin},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}

Acknowledgements

Built on top of the MONDAY dataset. We thank its authors for releasing the original data.