Yushi123/Gui-agent

Gui-Agent — GUI trajectories in LIBERO/VLA format

Human GUI demonstrations from four sources, unified into a single VLA-style intermediate representation and written as LIBERO-layout HDF5, so LIBERO/VLA dataloaders run against GUI data unchanged.

raw source ──[adapter]──> GuiEpisode ──[writer]──> LIBERO-style HDF5
              per-source   the IR         format-       what you train on
              only                        specific

25,872 episodes / 453,264 steps / 235 GB, web + desktop, screenshots at native source resolution.

⚠️ The training data is raw HDF5, not Parquet. Read it with h5py (see Loading); load_dataset() will not give you the 235 GB. The Dataset Viewer above shows a 665-row `preview` config — a sampled handful of steps, images downscaled, for browsing only. It is not the dataset.

The preview config

datasets has no HDF5 reader, so the viewer cannot render the shards at all (SplitsNotFoundError). That is ordinary for this format — `yifengzhu-hf/LIBERO-datasets`, whose layout this mirrors, has no viewer either — and it has no effect on h5py reads or hf download.

So preview/*.parquet carries 665 steps sampled across all nine directories, one row per step: the pre-action frame, the cursor crop, and the action taken from it, alongside the instruction and the mask. Scrolling it walks real trajectories rather than a table of metadata. Frames are JPEG at 640 px wide (eye-in-hand at 128); coordinates stay normalized, so they still line up.

from datasets import load_dataset
sample = load_dataset("Yushi123/Gui-agent", "preview", split="sample")  # 28 MB

Do not train on it — it is 665 steps of 453,264, resized and re-encoded.

Splits

Every number above is the main split only. The two AgentNet directories each carry an additional validation set, excluded from those counts — a 2% deterministic hash holdout, since AgentNet ships no official split:

Grand total across the repo: 491 shards, 25,872 episodes, 453,264 steps, 235 GB.

Shard layout

Files are named <directory>_NNNN.hdf5 (validation: <directory>_val_NNNN.hdf5) and each is capped at ~500 MB — most land at 490–501 MB, with one short shard at the tail of each series.

Every shard is a complete, self-contained HDF5. Open any one directly with h5py; there is nothing to concatenate and no part files. A shard boundary is just a demo boundary, so a directory's shards concatenate logically — iterate all of them to get the full split, in any order you like.

Shards are small because the upload link this was published over dropped every connection after a few minutes, which no single multi-GB file could survive. The size carries no meaning for training: pick shards, not episodes-per-shard, as your unit of parallelism and it makes no difference.

Each directory has a dataset_info.json recording the exact build config, the per-adapter stats (drops, scroll histograms, ungrounded-step counts), and the shard list with per-shard demo/step counts.

The POMDP convention — read this first

o_0 --a_0--> o_1 --a_1--> o_2  ...  o_{T-1} --a_{T-1}--> [o_T]

obs/agentview_rgb[t] is o_t: the screen as it looked _before_ a_t ran. It is the frame the policy conditions on to choose a_t.

Get this backwards — pair at with the frame showing at's effect — and you silently train a policy to predict the action it just watched happen. It looks excellent offline and does nothing online. Every adapter documents how it verified this alignment against its source.

The terminal frame o_T exists only where the source recorded one.

Layout

/data                              attrs: env_name, env_args, problem_info, num_demos,
  /demo_0                                 total, tag, gui_action_types, gui_param_names
    actions        (T, 7)  float32  LIBERO-shaped vector          [compat only]
    action_type    (T,)    int64    <- the real supervision
    action_params  (T, 4)  float32  <- the real supervision
    action_mask    (T, 4)  float32  <- masks the coord loss
    action_text    (T,)    str
    action_desc    (T,)    str      "click(0.71, 0.78)"
    dones          (T,)    uint8
    rewards        (T,)    uint8    sparse: 1 on the last step iff success
    states         (T, 5)  float32  [cursor_x, cursor_y, scroll_x, scroll_y, progress]
    robot_states   (T, 2)  float32  cursor
    eye_rect       (T, 4)  float32  where the eye-in-hand window sat, normalized
    native_hw      (T, 2)  int32    source resolution the coords were computed vs
    obs/
      agentview_rgb    (T, H, W, 3) uint8   the viewport, native resolution
      eye_in_hand_rgb  (T, 128, 128, 3) uint8
      cursor_states    (T, 2)  float32
      scroll_states    (T, 2)  float32   viewport offset within the page
      gripper_states   (T, 2)  float32   [compat] alias of cursor
      joint_states     (T, 7)  float32   [compat] zeros — a GUI has no arm

Three deviations from LIBERO that will bite you

1. 1.The instruction is per-demo, not per-file. A LIBERO file is one task × 50 demos, so its instruction lives in /data.attrs.problem_info. Every GUI episode is its own task — read demo_i.attrs["language_instruction"]. The file-level problem_info is written (loaders read it blindly) but its language_instruction is empty.

1. 1.`actions` is not a regression target. It is [x1, y1, x2, y2, action_type, has_text, n_masked], padded to width 7 purely so code hardcoding LIBERO's action width keeps running. Slot 4 is a category, not a magnitude — an MSE over this vector is meaningless. Train on action_type (cross-entropy) + action_params (regression, masked by action_mask).

1. 1.`joint_states` is zeros and gripper_states copies the cursor. They exist so proprio-shaped plumbing doesn't crash. Don't feed a model zeros.

Action space

action_type    : ActionType     CLICK TYPE SELECT HOVER SCROLL DRAG
                                PRESS_{BACK,HOME,ENTER,KEY} GOTO WAIT STOP FAIL
action_params  : float32[4]     [x1, y1, x2, y2], normalized [0,1] vs agentview
action_mask    : float32[4]     which params this step actually constrains
action_text    : str            typed text / key / url

`action_mask` is the load-bearing piece. A HOME press has no click point; 6% of Mind2Web steps have no locatable element. Mask the coordinate loss with it — a policy must not be penalized for whatever it emits in a slot the demonstration never constrained. Fabricating a coordinate there would be inventing supervision.

Scroll params are the gesture; the label is the view. action_params stores the finger/pointer travel, because that is what the sources record and what a device replays. But "scroll down" means the view moves down, and to move the view down you drag the content up. action_desc reports the view direction, which is the inverse of the gesture in the params.

The two camera views

The eye-in-hand crop follows the cursor, never the action target. No GUI source records a pointer position, so it is derived causally: the cursor before a_t is wherever a_{t-1} left the pointer. Step 0 starts at screen center.

Centering the crop on at's *own* target would paint the answer into the observation — "click the middle of the eye-in-hand view" would become a near-perfect policy and every offline number would be fiction. A healthy `targetinsideeyeinhandrate sits around **0.11–0.20**; exactly **1.0** is what leakage looks like. eye_rect` records the window actually used, so the check is exact rather than estimated.

Coordinates are normalized, so they stay grounded under any (anisotropic) resize with zero bookkeeping.

Source-specific notes

Mind2Web — the observation is a real browser viewport, not a resized page. Mind2Web stores the whole scrollable page (1280 px wide, up to ~85,000 px tall). No agent sees that. The capture viewport was 1280×1080 (read off <html>'s own bounding_box_rect), and the full-page screenshot is a 1:1 CSS-pixel render, so rows [S, S+1080) are exactly the pixels shown at scroll offset S — slicing is the render, not an approximation of one. Scrolling is therefore emitted as genuine SCROLL steps (~22% of steps), which is also what gives Mind2Web the scroll action it otherwise lacks. Two limits: position: fixed/sticky elements were baked in where they sat, and scroll resets to 0 each step (the source records no scroll state). 6.4% of steps have empty pos_candidates — kept with the coordinate masked, because dropping the episode would cost 28.9% of episodes to fix a 6.4% problem.

AgentNet — actions are PyAutoGUI source code, parsed with ast, not regex (write(message=...) routinely contains quotes and newlines of its own). One step can be several statements and the combination is the action (moveTo+dragTo → DRAG, moveTo+scroll → SCROLL). Steps holding two real actions (dragTo+hotkey, click+click) are dropped, not guessed — counts are in dataset_info.json.

WebLINX is read from McGill-NLP/WebLINX-zipped's replay.json, not the packaged chat format (which is a preprocessed text rendering that drops event metadata). Navigator say turns are not actions and are skipped.

⚠️ WebArena — `webarena_human/` demonstrates the WebArena EVAL tasks. Training on <task_id> and then scoring that same task_id online is training on the test set. The task id and intent_template_id are in episode metadata precisely so an eval can hold them — or their whole template — out. Use it only with that holdout in place.

Splits are the benchmarks', not ours

Mind2Web ships train plus the three standard eval sets test_task (176), test_website (142), test_domain (687); WebLINX ships train/valid. Those are used as-is. A hash-based holdout is not wrong, but it is different, and every number produced against it is incomparable with published results — which defeats the point of using a benchmark. Only AgentNet, which ships no official split, gets a deterministic 2% hash holdout.

Loading

import h5py, numpy as np
from huggingface_hub import hf_hub_download

p = hf_hub_download("Yushi123/Gui-agent", "mind2web_train/mind2web_train_0000.hdf5",
                    repo_type="dataset")

with h5py.File(p, "r") as f:
    demo = f["data"]["demo_0"]
    print(demo.attrs["language_instruction"])
    obs   = demo["obs"]["agentview_rgb"][:]   # (T, H, W, 3) uint8 — o_t, PRE-action
    eye   = demo["obs"]["eye_in_hand_rgb"][:] # (T, 128, 128, 3)
    atype = demo["action_type"][:]            # (T,)   int64  -> cross-entropy
    prm   = demo["action_params"][:]          # (T, 4) float32 -> masked regression
    mask  = demo["action_mask"][:]            # (T, 4) float32
    print(demo["action_desc"][:5])

# masked coordinate loss — never regress an unconstrained slot
loss_xy = (((pred - prm) ** 2) * mask).sum() / np.maximum(mask.sum(), 1)

Iterate a whole split — shards concatenate, demo_i restarts at 0 in each:

import glob
for shard in sorted(glob.glob("gui-agent/mind2web_train/*.hdf5")):
    with h5py.File(shard, "r") as f:
        for name in f["data"]:
            demo = f["data"][name]
            ...

Grab one directory instead of all 235 GB:

hf download Yushi123/Gui-agent --repo-type dataset \
    --include "mind2web_train/*" --local-dir ./gui-agent

# AgentNet train only, leaving its validation shards behind
hf download Yushi123/Gui-agent --repo-type dataset \
    --include "agentnet_ubuntu/agentnet_ubuntu_[0-9]*.hdf5" --local-dir ./gui-agent

The integer↔name mapping for action_type is on the file-level attribute /data.attrs["gui_action_types"].

Licensing and provenance

This repo redistributes derived renderings of four upstream datasets. Each retains its original terms — check the upstream before any downstream use; the license: other tag above reflects that the terms are mixed, not permissive by default.

Screenshots are of real websites and real desktops as captured by the upstream authors and may contain incidental third-party content; no additional filtering beyond upstream's was applied.

Please cite the upstream datasets, not just this repackaging.