CoolFace
Datasetpublic

pierretokns/screenspot-mlx-ccmcp

ScreenSpot MLX-VLM Dataset Multi-format VLM dataset for GUI grounding / pixel coordinate prediction. Formatted for training with mlx-vlm. Formats Each model family has its own data directory with the correct format: Model Family Directory Description SmolVLM/IDEFICS3 data/smolvlm/ Structured content blocks Qwen2-VL data/qwen2-vl/ Inline vision tokens PaliGemma data/paligemma/ Prefix token format LLaVA data/llava/ Newline separator format… See the full description on the dataset page: https://huggingface.co/datasets/pierretokns/screenspot-mlx-ccmcp.

sourceHugging Faceapache-2.0updated 8mo agoView on Hugging Face
0likes43downloads
Dataset Card

ScreenSpot MLX-VLM Dataset

Multi-format VLM dataset for GUI grounding / pixel coordinate prediction. Formatted for training with mlx-vlm.

Formats

Each model family has its own data directory with the correct format:

Model FamilyDirectoryDescription
SmolVLM/IDEFICS3data/smolvlm/Structured content blocks
Qwen2-VLdata/qwen2-vl/Inline vision tokens
PaliGemmadata/paligemma/Prefix token format
LLaVAdata/llava/Newline separator format

Statistics

  • —Total Examples: 1,272
  • —Train: 1,017
  • —Validation: 127
  • —Test: 128

Source Datasets

SourceHuggingFace IDExamples
ScreenSpotKevinQHLin/ScreenSpot1,272

Usage

With HuggingFace Datasets

python
from datasets import load_dataset

# Load SmolVLM format
ds = load_dataset("pierretokns/screenspot-mlx-ccmcp", data_dir="data/smolvlm")

# Load Qwen2-VL format
ds = load_dataset("pierretokns/screenspot-mlx-ccmcp", data_dir="data/qwen2-vl")

With mlx-vlm

bash
# Train SmolVLM
uv run python -m mlx_vlm.lora \
    --model HuggingFaceTB/SmolVLM-256M-Instruct \
    --data pierretokns/screenspot-mlx-ccmcp \
    --data-dir data/smolvlm \
    --iters 500

Output Format

All formats output JSON with action coordinates:

json
{"action": "left_click", "coordinate": [847, 523]}

SmolVLM Format

json
{"messages": [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "Click Submit"}]}, {"role": "assistant", "content": [{"type": "text", "text": "{...}"}]}], "images": ["images/001.png"]}

Qwen2-VL Format

json
{"messages": [{"role": "user", "content": "<|vision_start|><|image_pad|><|vision_end|>Click Submit"}, {"role": "assistant", "content": "{...}"}], "images": ["images/001.png"]}

License

Apache 2.0

Citation

If you use this dataset, please cite the original sources: