CoolFace
Datasetpublic

ScaleAI/VisualToolBench

VisToolBench Dataset A benchmark dataset for evaluating vision-language models on tool-use tasks. Dataset Statistics Total samples: 1204 Single-turn: 603 Multi-turn: 601 Schema Column Type Description id string Unique task identifier turncase string Either "single-turn" or "multi-turn" num_turns int Number of conversation turns (1 for single-turn) prompt_category string Task category (e.g., "medical", "scientific", "general")… See the full description on the dataset page: https://huggingface.co/datasets/ScaleAI/VisualToolBench.

sourceHugging Faceupdated 9mo agoView on Hugging Face
7likes614downloads
Dataset Card

VisToolBench Dataset

A benchmark dataset for evaluating vision-language models on tool-use tasks.

Dataset Statistics

  • —Total samples: 1204
  • —Single-turn: 603
  • —Multi-turn: 601

Schema

ColumnTypeDescription
idstringUnique task identifier
turncasestringEither "single-turn" or "multi-turn"
num_turnsintNumber of conversation turns (1 for single-turn)
prompt_categorystringTask category (e.g., "medical", "scientific", "general")
eval_focusstringWhat aspect is being evaluated (e.g., "visualreasoning", "tooluse")
turn_promptsList[string]Per-turn prompts (single-turn → list of length 1)
turn_golden_answersList[string]Per-turn golden answers
turn_tool_trajectoriesList[string]Per-turn tool trajectories (JSON strings)
rubrics_by_turnList[string]Per-turn rubric dicts as JSON strings (includes weights + metadata)
imagesList[Image]Flat list of all images (HF viewer shows these)
images_by_turnList[List[Image]]Images grouped by turn (to know which image belongs to which turn)
num_imagesintTotal images in images

Rubrics Format

Each rubric entry contains:

  • —description: What the rubric evaluates
  • —weight: Importance weight (1-5)
  • —objective/subjective: Whether evaluation is objective or subjective
  • —explicit/implicit: Whether the answer is explicit or implicit in the image
  • —category: List of categories (e.g., "instruction following", "truthfulness")
  • —critical: Whether this is a critical rubric ("yes"/"no")
  • —final_answer: Whether this relates to the final answer ("yes"/"no")

Usage

python
from datasets import load_dataset

# Load the dataset
ds = load_dataset("path/to/dataset")

# Access a sample
sample = ds['test'][0]
print(sample['turn_prompts'])       # list[str]
print(sample['images'][0])          # PIL Image (first image overall)
print(sample['images_by_turn'][0])  # list of PIL Images for turn 1

# Parse rubrics for turn 1
import json
turn1_rubrics = json.loads(sample['rubrics_by_turn'][0])
for rubric_id, rubric in turn1_rubrics.items():
    print(f"{rubric['description']} (weight: {rubric['weight']})")

Splits

  • —test: Full dataset (1204 samples)

Citation

bibtex
@article{guo2025beyond,
  title={Beyond seeing: Evaluating multimodal llms on tool-enabled image perception, transformation, and reasoning},
  author={Guo, Xingang and Tyagi, Utkarsh and Gosai, Advait and Vergara, Paula and Park, Jayeon and Montoya, Ernesto Gabriel Hern{\'a}ndez and Zhang, Chen Bo Calvin and Hu, Bin and He, Yunzhong and Liu, Bing and others},
  journal={arXiv preprint arXiv:2510.12712},
  year={2025}
}