WarmwindOS/pointerbench

Pointerbench

Pointerbench is a small GUI grounding benchmark suite for computer-use models. Each example has one screenshot, one instruction, target geometry in absolute pixels, and a binary evaluation rule.

Links:

• —GitHub: https://github.com/warmwindOS/pointerbench
• —Blog post: https://about.warmwind.com/pointer-bench/
• —Add your model to the official benchmark leaderboard: https://warmwind.com/contact

The suite has three subsets:

Leaderboard

Scores are percentages. The table is ranked by the reported public average across Sheets, Text, and Pro.

To add your model to the official benchmark leaderboard, contact https://warmwind.com/contact.

All images are synthetic 1024x768 PNG screenshots. The datasets contain no scraped user data and no PII.

Layout

Each subset is self-contained:

pointerbench-sheets/
  data/test/metadata.jsonl
  data/test/0000.png
  eval.py
  README.md
  REPRODUCE.md

pointerbench-text/
  data/test/metadata.jsonl
  data/test/0000.png
  eval.py
  README.md
  REPRODUCE.md

pointerbench-pro/
  data/test/metadata.jsonl
  data/test/0000.png
  eval.py
  README.md
  REPRODUCE.md

Schema

Each metadata row includes:

{
  "file_name": "0000.png",
  "id": "pbs_0000",
  "instruction": "Click cell E11.",
  "bbox": [x1, y1, x2, y2],
  "point": [x, y],
  "answer_type": "point",
  "eval": {"type": "point_in_bbox", "bbox": [x1, y1, x2, y2]},
  "data_type": "cell",
  "category": "cell_ref",
  "image_size": [1024, 768]
}

Point tasks are correct when the predicted point lands inside the target bbox. Bbox tasks, used in Pointerbench-Text, use an asymmetric overlap rule: a hit requires the ground truth to be almost fully covered (coverage >= 0.90) and the prediction to stay reasonably tight around it (precision >= 0.70). This penalises predictions that cut off part of the target far more than predictions that wrap it with some margin.

Evaluation

Run the scorer inside a subset folder:

python eval.py --predictions preds.jsonl

Predictions are JSONL rows with an id and either a point or bbox, depending on answer_type.

Recommended inference prompt:

python eval.py --show-system-prompt

You are evaluating Pointerbench, a GUI grounding benchmark. You will receive one 1024x768 screenshot and one task instruction. Use absolute pixel coordinates with origin at the top-left of the image. Do not return normalized coordinates. Do not crop or resize the coordinate frame. For point tasks, return JSON like {"point": [x, y]}. For bounding-box tasks, return JSON like {"bbox": [x0, y0, x1, y1]}.

You can edit the prompt for your inference stack. Keep the 1024x768 absolute pixel coordinate frame fixed, and report any image resizing or multi-step zoom strategy with your results.

See each subset README for the exact distribution, schema details, and examples.

License

Dataset images and annotations are released under CC BY 4.0. The included evaluation scripts are released under MIT.