cua-lite/Multimodal-Mind2Web

cua-lite/Multimodal-Mind2Web

cua-lite preprocessed version of Multimodal-Mind2Web (osunlp/Multimodal-Mind2Web). The screenshot-augmented Mind2Web web-agent benchmark: real human-annotated multi-step navigation episodes across 100+ websites. Only the train split is processed (the testtask/testwebsite/test_domain benchmark holdouts are never trained on). Episodes containing a native <select> step are dropped (no native select tool in the cua-lite action space).

Origin

• —https://huggingface.co/datasets/osunlp/Multimodal-Mind2Web

Load via datasets

from datasets import load_dataset

# entire dataset
ds = load_dataset("cua-lite/Multimodal-Mind2Web")

# just one (platform, task_type) cohort
ds = load_dataset("cua-lite/Multimodal-Mind2Web", "browser.use")

After loading, parse metadata as JSON before filtering by metadata_kind, dims, or others.*; every row carries a rich metadata object inside that JSON string (see schema below). CUA rows use dims == [platform, task_type].

Schema

Published parquet columns:

Coordinate values in messages are normalized to [0, 1000] integers. The JSON examples below show the decoded shape, not the raw string cell.

metadata.extra_tool_schemas[*] uses the nested Chat Completions function-tool declaration shape:

{
  "metadata_kind": "cua",
  "dims": ["desktop", "use"],
  "extra_tool_schemas": [
    {
      "type": "function",
      "function": {
        "name": "bash",
        "description": "Run a shell command.",
        "parameters": {
          "type": "object",
          "properties": {"cmd": {"type": "string"}},
          "required": ["cmd"]
        }
      }
    }
  ],
  "valid_actions": ["click", "type"],
  "others": {}
}

messages[].tool_calls[*] uses the matching nested invocation shape. Tool results pair tool_calls[].id with role:"tool" tool_call_id:

[
  {
    "role": "user",
    "content": [
      {"type": "image", "index": 0},
      {"type": "text", "text": "Click the OK button."}
    ]
  },
  {
    "role": "assistant",
    "tool_calls": [
      {
        "id": "call_0000",
        "type": "function",
        "function": {
          "name": "computer",
          "arguments": {
            "actions": [
              {"action": "click", "coordinate": [640, 400]},
              {"action": "type", "text": "hello"}
            ]
          }
        }
      }
    ]
  },
  {
    "role": "tool",
    "tool_call_id": "call_0000",
    "content": [
      {"type": "image", "index": 1},
      {"type": "text", "text": "clicked; typed"}
    ]
  }
]

*Image-dedup (`grounding. / understanding cohorts).** These cohorts are single-image-per-row and many rows share the same screenshot, so to avoid re-embedding identical image bytes once per instruction they are stored *folded*: one row per unique screenshot (image embedded once), carrying an extra **_folded** column — a JSON string with the authoritative list of per-instruction members for that screenshot. Each member's messages and metadata values are the same opaque JSON strings described above. The row's top-level messages is a JSON string containing the members concatenated for viewer convenience. use cohorts are not folded. **Use lite.data.hf.download` to consume this repo** — it unfolds automatically back to one row per instruction; reading the parquet directly yields the folded form.

Layout

<platform>/<task_type>/<split>/<variant>/shard-NNNNN-of-NNNNN.parquet

• —platform ∈ {desktop, browser, mobile}
• —task_type ∈ {understanding, grounding.action, grounding.point, grounding.bbox, use} — used verbatim as the dir component
• —HF config names are <platform>.<task_type> by default (e.g. mobile.grounding.action) — UNLESS the dataset was staged with --config-names, which sets verbatim, explicitly-chosen config names (see the configs: block above for the authoritative list). The agent registry lookup key in code is <agent>@<platform>@<task_type> (e.g. qwen3_vl@mobile@grounding.action); only this user-facing token uses . between platform and task_type, because @ triggers a 403 on the dataset-viewer's signed image URLs.
• —HF split names stay train / validation (the datasets library blacklists <>:/\|?* in split names; everything else is fine in config_name)
• —validation is an in-distribution held-out slice: no validation sample also appears in train — content-identical rows (same images + same messages, differing only in their ids) are co-located into one split, so upstream re-publishing one sample under two ids cannot straddle the split. It is not disjoint in images: one screenshot legitimately backs many distinct samples, and only whole samples are co-located, so the same picture can appear on both sides. test is reserved for out-of-distribution benchmark datasets

Stats

Local mirror & SFT export

For local workflows (SFT export, dedup, mixing across datasets), use lite.data.hf.download to mirror this repo back to the canonical local layout:

$CUA_LITE_DATASETS_ROOT/cua-lite/Multimodal-Mind2Web/
  images/<hash[:2]>/<hash>.<ext>                          # content-addressed image store
  <platform>/<task_type>/<split>[/<variant>].parquet      # rows reference images by relative path

Rows in the local parquet have images: list[str]; bytes are extracted to the image store. lite.train.export.export_sft consumes the local form directly with --image-root=$CUA_LITE_DATASETS_ROOT.

• —Total unique images: 3,803
• —Image store size: 3.68 GB

Notes

Train split only; test_* splits are benchmark holdouts and are not produced. Coordinates are the centre of the target element's bounding box, normalized to [0,1000].

split: content-identical rows co-located, then hashsplit on metadata.others.id with valfrac=0.02, seed=42, val_cap=2000 per (metadata.dims[0], metadata.dims[1], variant) (an upstream split label, where the source ships one, wins over the hash). A cap-bound carve depends on source iteration order and these parameters do not reproduce it; content co-location may make the final physical validation row count differ from the cap.

License & citation

See original dataset (osunlp/Multimodal-Mind2Web).

See https://huggingface.co/datasets/osunlp/Multimodal-Mind2Web