ScaleAI/VisualToolBench

1---2pretty_name: VisualToolBench3tags:4  - vision5  - multimodal6  - tool-use7task_categories:8  - visual-question-answering9---10 11# VisToolBench Dataset12 13A benchmark dataset for evaluating vision-language models on tool-use tasks.14 15## Dataset Statistics16 17- **Total samples**: 120418- **Single-turn**: 60319- **Multi-turn**: 60120 21## Schema22 23| Column | Type | Description |24|--------|------|-------------|25| `id` | string | Unique task identifier |26| `turncase` | string | Either "single-turn" or "multi-turn" |27| `num_turns` | int | Number of conversation turns (1 for single-turn) |28| `prompt_category` | string | Task category (e.g., "medical", "scientific", "general") |29| `eval_focus` | string | What aspect is being evaluated (e.g., "visual_reasoning", "tool_use") |30| `turn_prompts` | List[string] | Per-turn prompts (single-turn → list of length 1) |31| `turn_golden_answers` | List[string] | Per-turn golden answers |32| `turn_tool_trajectories` | List[string] | Per-turn tool trajectories (JSON strings) |33| `rubrics_by_turn` | List[string] | Per-turn rubric dicts as JSON strings (includes weights + metadata) |34| `images` | List[Image] | Flat list of all images (HF viewer shows these) |35| `images_by_turn` | List[List[Image]] | Images grouped by turn (to know which image belongs to which turn) |36| `num_images` | int | Total images in `images` |37 38## Rubrics Format39 40Each rubric entry contains:41- `description`: What the rubric evaluates42- `weight`: Importance weight (1-5)43- `objective/subjective`: Whether evaluation is objective or subjective44- `explicit/implicit`: Whether the answer is explicit or implicit in the image45- `category`: List of categories (e.g., "instruction following", "truthfulness")46- `critical`: Whether this is a critical rubric ("yes"/"no")47- `final_answer`: Whether this relates to the final answer ("yes"/"no")48 49## Usage50 51```python52from datasets import load_dataset53 54# Load the dataset55ds = load_dataset("path/to/dataset")56 57# Access a sample58sample = ds['test'][0]59print(sample['turn_prompts'])       # list[str]60print(sample['images'][0])          # PIL Image (first image overall)61print(sample['images_by_turn'][0])  # list of PIL Images for turn 162 63# Parse rubrics for turn 164import json65turn1_rubrics = json.loads(sample['rubrics_by_turn'][0])66for rubric_id, rubric in turn1_rubrics.items():67    print(f"{rubric['description']} (weight: {rubric['weight']})")68```69 70## Splits71 72- `test`: Full dataset (1204 samples)73 74## Citation75 76```bibtex77@article{guo2025beyond,78  title={Beyond seeing: Evaluating multimodal llms on tool-enabled image perception, transformation, and reasoning},79  author={Guo, Xingang and Tyagi, Utkarsh and Gosai, Advait and Vergara, Paula and Park, Jayeon and Montoya, Ernesto Gabriel Hern{\'a}ndez and Zhang, Chen Bo Calvin and Hu, Bin and He, Yunzhong and Liu, Bing and others},80  journal={arXiv preprint arXiv:2510.12712},81  year={2025}82}83```84